2024-03-04T15:46:07.616 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-03-04T15:46:07.616 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_tonay@teuthology --name tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} --archive /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239 -- /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/orig.config.yaml 2024-03-04T15:46:07.649 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T15:46:07.771 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239 branch: wip-asm0deuz-testing-2024-03-04-1211 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: tonay@ibm.com first_in_suite: false job_id: '7581239' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-asm0deuz-testing-2024-03-04-1211 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-asm0deuz-testing-2024-03-04-1211 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 owner: scheduled_tonay@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 3905 sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 sleep_before_teardown: 0 subset: 1/14 suite: orch:cephadm suite_branch: wip-asm0deuz-testing-2024-03-04-1211 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWPHYoC11lbobcDd3OU4DfFN73nuWEPHzYCvyheTmedIbav+2IEE5DJFEM2LleEfu0glEDyCpNsSmbv82GUOJ4= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7TncKNCLhunUYsBSAMR0akyNySygcv1hw1IEgYtU9a4jDPcms819RdzNhFYLD7lXC6gyUKeoY3JFyhShtiDQk= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-03-04_15:07:38 tube: smithi user: tonay verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-04T15:46:07.771 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa; will attempt to use it 2024-03-04T15:46:07.772 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9ca0b5de35ed230e84387abdd8e619ed8e4f3282/qa/tasks 2024-03-04T15:46:07.773 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-04T15:46:07.775 INFO:teuthology.task.internal:Checking packages... 2024-03-04T15:46:07.810 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9ca0b5de35ed230e84387abdd8e619ed8e4f3282' 2024-03-04T15:46:07.810 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-04T15:46:07.810 INFO:teuthology.packaging:ref: None 2024-03-04T15:46:07.810 INFO:teuthology.packaging:tag: None 2024-03-04T15:46:07.810 INFO:teuthology.packaging:branch: wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:46:07.811 INFO:teuthology.packaging:sha1: 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:46:07.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-asm0deuz-testing-2024-03-04-1211 2024-03-04T15:46:07.992 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1826.g9ca0b5de 2024-03-04T15:46:07.994 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-04T15:46:08.001 INFO:teuthology.task.internal:no buildpackages task found 2024-03-04T15:46:08.001 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-04T15:46:08.013 INFO:teuthology.task.internal:Saving configuration 2024-03-04T15:46:08.026 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-04T15:46:08.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T15:46:08.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:41:03.192480', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWPHYoC11lbobcDd3OU4DfFN73nuWEPHzYCvyheTmedIbav+2IEE5DJFEM2LleEfu0glEDyCpNsSmbv82GUOJ4='} 2024-03-04T15:46:08.139 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:41:03.194042', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7TncKNCLhunUYsBSAMR0akyNySygcv1hw1IEgYtU9a4jDPcms819RdzNhFYLD7lXC6gyUKeoY3JFyhShtiDQk='} 2024-03-04T15:46:08.140 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-04T15:46:08.146 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-04T15:46:08.146 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-04T15:46:08.146 INFO:teuthology.run_tasks:Running task console_log... 2024-03-04T15:46:08.272 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb010aeaf70>, signals=[15]) 2024-03-04T15:46:08.272 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-04T15:46:08.279 INFO:teuthology.task.internal:Opening connections... 2024-03-04T15:46:08.279 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-03-04T15:46:08.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:46:08.426 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-03-04T15:46:08.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:46:08.521 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-04T15:46:08.530 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-03-04T15:46:08.550 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-03-04T15:46:08.550 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:46:08.607 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:46:08.608 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:46:08.608 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:46:08.608 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-03-04T15:46:08.640 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-03-04T15:46:08.656 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-03-04T15:46:08.657 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-03-04T15:46:08.717 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-04T15:46:08.718 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-04T15:46:08.718 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-03-04T15:46:08.748 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-04T15:46:08.758 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-04T15:46:08.762 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-04T15:46:08.763 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:46:08.765 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-03-04T15:46:08.781 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-04T15:46:08.786 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-04T15:46:08.786 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:46:08.823 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-03-04T15:46:08.849 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-04T15:46:08.925 INFO:teuthology.run_tasks:Running task kernel... 2024-03-04T15:46:08.940 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-04T15:46:08.941 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-04T15:46:08.941 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:46:08.942 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-04T15:46:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:46:08.957 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-04T15:46:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:46:08.958 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-04T15:46:09.015 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:46:09.016 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-361.el9.x86_64 2024-03-04T15:46:09.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-03-04T15:46:09.017 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-03-04T15:46:09.017 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-03-04T15:46:09.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-03-04T15:46:11.082 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-04T15:46:11.121 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-04T15:46:12.524 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-04T15:46:12.630 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-04T15:46:13.457 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-03-04T15:46:13.542 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-04T15:46:15.408 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-04T15:46:17.272 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-03-04T15:46:19.369 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-04T15:46:21.492 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-04T15:46:21.659 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 15 kB 00:01 2024-03-04T15:46:22.902 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 18 MB/s | 20 MB 00:01 2024-03-04T15:46:22.966 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 15 kB 00:00 2024-03-04T15:46:26.028 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 20 MB 00:02 2024-03-04T15:46:27.743 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-04T15:46:29.984 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:46:29.985 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:46:30.066 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:46:30.073 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:46:30.073 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-04T15:46:30.073 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-03-04T15:46:30.074 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:46:30.075 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 100 M 2024-03-04T15:46:30.075 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-03-04T15:46:30.075 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:46:30.603 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 22 MB/s | 6.0 MB 00:00 2024-03-04T15:46:30.887 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 38 MB/s | 21 MB 00:00 2024-03-04T15:46:31.134 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-04T15:46:31.254 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 52 MB/s | 33 MB 00:00 2024-03-04T15:46:31.620 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 31 MB/s | 39 MB 00:01 2024-03-04T15:46:31.624 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:46:31.624 INFO:teuthology.orchestra.run.smithi176.stdout:Total 64 MB/s | 100 MB 00:01 2024-03-04T15:46:32.514 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:46:32.906 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:46:32.906 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:46:33.432 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-04T15:46:33.433 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-04T15:46:33.533 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-04T15:46:33.539 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:46:33.540 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 100 M 2024-03-04T15:46:33.541 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 123 M 2024-03-04T15:46:33.541 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:46:33.844 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 23 MB/s | 6.0 MB 00:00 2024-03-04T15:46:34.136 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 38 MB/s | 21 MB 00:00 2024-03-04T15:46:34.238 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:46:34.238 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:46:34.536 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 49 MB/s | 33 MB 00:00 2024-03-04T15:46:34.936 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 29 MB/s | 39 MB 00:01 2024-03-04T15:46:34.939 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:46:34.940 INFO:teuthology.orchestra.run.smithi038.stdout:Total 71 MB/s | 100 MB 00:01 2024-03-04T15:46:35.858 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:46:36.320 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:46:36.320 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:46:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:46:37.564 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:46:37.695 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:46:37.793 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:46:37.793 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:46:40.514 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:46:40.644 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:46:40.695 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:46:41.337 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:46:41.480 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:46:44.427 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:46:44.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:46:45.409 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:46:45.538 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:46:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:46:49.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:46:50.321 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:46:54.957 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:19.635 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:47:19.635 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:47:19.720 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:20.182 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:20.182 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:47:20.182 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:47:20.183 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:47:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:47:20.580 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:47:20.580 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:47:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:47:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:47:20.581 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:47:20.581 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:47:21.096 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:47:21.249 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:21.250 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:47:21.250 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:47:21.250 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:47:21.250 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:47:21.250 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-04T15:47:21.250 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:47:21.251 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:47:21.372 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:21.372 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:47:21.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-03-04T15:47:21.406 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:47:21.407 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:47:21.408 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:47:21.408 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:47:21.408 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-03-04T15:47:21.408 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:47:21.408 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:47:21.409 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:47:21.409 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:47:21.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:47:22.017 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-03-04T15:47:23.378 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:47:23.408 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-03-04T15:47:23.410 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-03-04T15:47:23.425 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.aGwEPPTZ5f 2024-03-04T15:47:23.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aGwEPPTZ5f 2024-03-04T15:47:23.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.aGwEPPTZ5f 2024-03-04T15:47:23.630 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.aGwEPPTZ5f is 6KB 2024-03-04T15:47:23.679 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.aGwEPPTZ5f 2024-03-04T15:47:23.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:47:23.758 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:47:23.758 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:47:23.758 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:47:23.758 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:47:23.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:47:23.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-03-04T15:47:26.099 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:26.099 INFO:teuthology.orchestra.run.smithi038.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-04T15:47:26.099 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:47:26.192 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:26.635 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:26.636 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-04T15:47:26.636 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-04T15:47:26.636 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-04T15:47:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-04T15:47:27.061 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:47:27.061 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:47:27.062 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:47:27.548 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:47:27.749 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.749 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:47:27.749 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-04T15:47:27.750 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-04T15:47:27.750 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-04T15:47:27.750 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-04T15:47:27.750 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-03-04T15:47:27.750 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-04T15:47:27.875 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-04T15:47:27.875 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-04T15:47:27.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-03-04T15:47:27.922 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-03-04T15:47:27.922 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-03-04T15:47:27.922 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-03-04T15:47:27.922 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 68.el9 2024-03-04T15:47:27.922 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8264254 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-03-04T15:47:27.923 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-03-04T15:47:27.924 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:47:27.925 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-03-04T15:47:27.925 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-04T15:47:27.926 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-427.el9.x86_64 2024-03-04T15:47:27.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-04T15:47:28.476 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-03-04T15:47:30.003 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-04T15:47:30.038 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-03-04T15:47:30.040 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-04T15:47:30.095 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.xRGtz8tw3B 2024-03-04T15:47:30.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.xRGtz8tw3B 2024-03-04T15:47:30.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.xRGtz8tw3B 2024-03-04T15:47:30.316 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.xRGtz8tw3B is 6KB 2024-03-04T15:47:30.364 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.xRGtz8tw3B 2024-03-04T15:47:30.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-03-04T15:47:30.447 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-04T15:47:30.448 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-04T15:47:30.448 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-04T15:47:30.448 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-04T15:47:30.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-04T15:47:30.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-03-04T15:47:53.932 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-04T15:47:53.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:47:53.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:48:00.612 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-04T15:48:00.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:48:00.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:48:12.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-04T15:48:19.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-04T15:48:21.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:48:21.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:48:24.636 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-04T15:48:28.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:48:28.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:48:36.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:48:36.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:48:54.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-03-04T15:49:09.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:49:09.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:49:10.368 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-04T15:49:10.792 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:49:10.792 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:49:10.792 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-03-04T15:49:10.806 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:49:10.806 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:49:10.806 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:49:10.806 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:49:11.808 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-04T15:49:11.808 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:49:11.895 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-03-04T15:49:11.948 DEBUG:teuthology.parallel:result is None 2024-03-04T15:49:28.178 DEBUG:teuthology.orchestra.remote:timed out 2024-03-04T15:49:40.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:49:40.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:49:40.531 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-04T15:49:40.905 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:49:40.906 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-04T15:49:40.906 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-04T15:49:40.920 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-427.el9.x86_64 2024-03-04T15:49:40.920 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-04T15:49:40.921 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-04T15:49:40.921 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-04T15:49:41.921 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-04T15:49:41.921 DEBUG:teuthology.orchestra.run.smithi038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-04T15:49:42.004 INFO:teuthology.orchestra.run.smithi038.stdout:ttyS1 2024-03-04T15:49:42.040 DEBUG:teuthology.parallel:result is None 2024-03-04T15:49:42.040 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-04T15:49:42.048 INFO:teuthology.task.internal:Creating test directory... 2024-03-04T15:49:42.048 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:49:42.050 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-04T15:49:42.069 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-04T15:49:42.077 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-04T15:49:42.086 INFO:teuthology.task.internal:Creating archive directory... 2024-03-04T15:49:42.086 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:49:42.107 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-04T15:49:42.180 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-04T15:49:42.187 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-04T15:49:42.188 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:49:42.190 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-04T15:49:42.221 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:49:42.223 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:49:42.235 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:49:42.237 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-04T15:49:42.238 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-04T15:49:42.246 INFO:teuthology.task.internal:Configuring sudo... 2024-03-04T15:49:42.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:49:42.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-04T15:49:42.309 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-04T15:49:42.319 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-04T15:49:42.319 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:49:42.348 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-04T15:49:42.365 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:49:42.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:49:42.494 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:49:42.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:49:42.617 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:49:42.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:49:42.684 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:49:42.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-04T15:49:42.781 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:49:42.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-04T15:49:42.908 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:49:42.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-04T15:49:42.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-03-04T15:49:42.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-03-04T15:49:43.020 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:49:43.119 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T15:49:43.526 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-04T15:49:43.534 INFO:teuthology.task.internal:Starting timer... 2024-03-04T15:49:43.534 INFO:teuthology.run_tasks:Running task pcp... 2024-03-04T15:49:43.547 INFO:teuthology.run_tasks:Running task selinux... 2024-03-04T15:49:43.565 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-04T15:49:43.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-03-04T15:49:43.612 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-03-04T15:49:43.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-03-04T15:49:43.667 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-03-04T15:49:43.669 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-04T15:49:43.669 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-03-04T15:49:43.696 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-03-04T15:49:43.696 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-03-04T15:49:43.765 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-03-04T15:49:43.765 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-03-04T15:49:43.765 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:49:43.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:49:43.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T15:49:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T15:49:43.821 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-04T15:49:43.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:49:43.864 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-03-04T15:49:43.904 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-04T15:49:43.930 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-04T15:49:43.940 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-04T15:49:44.065 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-04T15:49:44.082 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-04T15:49:44.084 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-04T15:53:57.857 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-03-04T15:53:57.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:53:57.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:53:57.993 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-04T15:53:58.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-04T15:53:58.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:53:58.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-04T15:53:58.200 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-03-04T15:53:58.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-03-04T15:53:58.278 INFO:teuthology.run_tasks:Running task clock... 2024-03-04T15:53:58.289 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-04T15:53:58.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:53:58.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:53:58.292 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-04T15:53:58.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T15:53:58.321 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:53:58.338 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:53:58.348 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-04T15:53:58.364 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-04T15:53:58.368 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-03-04T15:53:58.381 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-03-04T15:53:58.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-03-04T15:53:58.398 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:53:58.407 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-03-04T15:53:58.414 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:53:58.423 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-04T15:53:58.439 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-04T15:53:58.471 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.475 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.489 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.493 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T15:53:58.493 INFO:teuthology.run_tasks:Running task install... 2024-03-04T15:53:58.502 DEBUG:teuthology.task.install:project ceph 2024-03-04T15:53:58.502 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'}} 2024-03-04T15:53:58.502 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'} 2024-03-04T15:53:58.502 INFO:teuthology.task.install:Using flavor: default 2024-03-04T15:53:58.508 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-04T15:53:58.508 INFO:teuthology.task.install:extra packages: [] 2024-03-04T15:53:58.508 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'tag': None, 'wait_for_package': False} 2024-03-04T15:53:58.509 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:53:58.511 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'tag': None, 'wait_for_package': False} 2024-03-04T15:53:58.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:53:58.668 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/ 2024-03-04T15:53:58.669 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1826.g9ca0b5de 2024-03-04T15:53:58.800 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T15:53:58.800 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:53:58.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T15:53:58.805 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/ 2024-03-04T15:53:58.806 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1826.g9ca0b5de 2024-03-04T15:53:58.829 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T15:53:58.829 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T15:53:58.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T15:53:58.977 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T15:53:58.995 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-04T15:53:58.995 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:53:58.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-04T15:53:59.024 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-04T15:53:59.024 DEBUG:teuthology.orchestra.run.smithi176:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-04T15:53:59.043 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-03-04T15:53:59.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-03-04T15:53:59.093 DEBUG:teuthology.orchestra.run.smithi176:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-04T15:53:59.173 DEBUG:teuthology.orchestra.run.smithi176:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-04T15:53:59.240 INFO:teuthology.orchestra.run.smithi176.stdout:check_obsoletes = 1 2024-03-04T15:53:59.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum clean all 2024-03-04T15:53:59.291 INFO:teuthology.orchestra.run.smithi038.stdout:74 files removed 2024-03-04T15:53:59.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw 2024-03-04T15:53:59.484 INFO:teuthology.orchestra.run.smithi176.stdout:74 files removed 2024-03-04T15:53:59.512 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-radosgw 2024-03-04T15:54:00.110 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 174 kB/s | 82 kB 00:00 2024-03-04T15:54:00.278 INFO:teuthology.orchestra.run.smithi176.stdout:ceph packages for x86_64 183 kB/s | 82 kB 00:00 2024-03-04T15:54:00.694 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-04T15:54:00.753 INFO:teuthology.orchestra.run.smithi176.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-04T15:54:01.245 INFO:teuthology.orchestra.run.smithi176.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-03-04T15:54:01.320 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-04T15:54:01.947 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-04T15:54:03.015 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 4.1 MB/s | 6.7 MB 00:01 2024-03-04T15:54:03.539 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-04T15:54:04.374 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-04T15:54:04.631 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 4.4 MB/s | 2.0 MB 00:00 2024-03-04T15:54:05.832 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.0 MB 00:00 2024-03-04T15:54:10.464 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-04T15:54:15.001 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-04T15:54:16.624 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-04T15:54:20.109 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 20 MB 00:03 2024-03-04T15:54:25.301 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-04T15:54:27.430 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 894 kB/s | 19 MB 00:21 2024-03-04T15:54:27.982 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 9.8 M 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.4 M 2024-03-04T15:54:27.986 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.1 M 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 5.3 M 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 20 M 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 25 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 745 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 501 k 2024-03-04T15:54:27.987 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 5.1 M 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 45 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 131 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 161 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 320 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 298 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 99 k 2024-03-04T15:54:27.988 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout:Install 30 Packages 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 2 Packages 2024-03-04T15:54:27.989 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:27.990 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 58 M 2024-03-04T15:54:27.990 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:54:29.168 INFO:teuthology.orchestra.run.smithi038.stdout:(1/32): ceph-base-19.0.0-1826.g9ca0b5de.el9.x86 6.5 MB/s | 5.3 MB 00:00 2024-03-04T15:54:29.210 INFO:teuthology.orchestra.run.smithi038.stdout:(2/32): ceph-selinux-19.0.0-1826.g9ca0b5de.el9. 604 kB/s | 25 kB 00:00 2024-03-04T15:54:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:(3/32): libcephfs2-19.0.0-1826.g9ca0b5de.el9.x8 8.7 MB/s | 745 kB 00:00 2024-03-04T15:54:29.369 INFO:teuthology.orchestra.run.smithi038.stdout:(4/32): libradosstriper1-19.0.0-1826.g9ca0b5de. 6.6 MB/s | 501 kB 00:00 2024-03-04T15:54:29.745 INFO:teuthology.orchestra.run.smithi038.stdout:(5/32): librgw2-19.0.0-1826.g9ca0b5de.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-04T15:54:29.787 INFO:teuthology.orchestra.run.smithi038.stdout:(6/32): python3-ceph-argparse-19.0.0-1826.g9ca0 1.1 MB/s | 45 kB 00:00 2024-03-04T15:54:29.837 INFO:teuthology.orchestra.run.smithi038.stdout:(7/32): python3-ceph-common-19.0.0-1826.g9ca0b5 2.6 MB/s | 131 kB 00:00 2024-03-04T15:54:29.887 INFO:teuthology.orchestra.run.smithi038.stdout:(8/32): python3-cephfs-19.0.0-1826.g9ca0b5de.el 3.1 MB/s | 161 kB 00:00 2024-03-04T15:54:29.962 INFO:teuthology.orchestra.run.smithi038.stdout:(9/32): python3-rados-19.0.0-1826.g9ca0b5de.el9 4.2 MB/s | 320 kB 00:00 2024-03-04T15:54:30.029 INFO:teuthology.orchestra.run.smithi038.stdout:(10/32): python3-rbd-19.0.0-1826.g9ca0b5de.el9. 4.4 MB/s | 298 kB 00:00 2024-03-04T15:54:30.088 INFO:teuthology.orchestra.run.smithi038.stdout:(11/32): python3-rgw-19.0.0-1826.g9ca0b5de.el9. 1.7 MB/s | 99 kB 00:00 2024-03-04T15:54:30.305 INFO:teuthology.orchestra.run.smithi038.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 489 kB/s | 106 kB 00:00 2024-03-04T15:54:30.405 INFO:teuthology.orchestra.run.smithi038.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-04T15:54:30.522 INFO:teuthology.orchestra.run.smithi038.stdout:(14/32): ceph-radosgw-19.0.0-1826.g9ca0b5de.el9 4.5 MB/s | 9.8 MB 00:02 2024-03-04T15:54:30.548 INFO:teuthology.orchestra.run.smithi038.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 310 kB/s | 44 kB 00:00 2024-03-04T15:54:30.623 INFO:teuthology.orchestra.run.smithi038.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-04T15:54:30.648 INFO:teuthology.orchestra.run.smithi038.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-03-04T15:54:30.673 INFO:teuthology.orchestra.run.smithi038.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-04T15:54:30.707 INFO:teuthology.orchestra.run.smithi038.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 3.6 MB/s | 215 kB 00:00 2024-03-04T15:54:30.732 INFO:teuthology.orchestra.run.smithi038.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 581 kB/s | 34 kB 00:00 2024-03-04T15:54:30.758 INFO:teuthology.orchestra.run.smithi038.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 865 kB/s | 43 kB 00:00 2024-03-04T15:54:30.975 INFO:teuthology.orchestra.run.smithi038.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-04T15:54:31.033 INFO:teuthology.orchestra.run.smithi038.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 448 kB/s | 26 kB 00:00 2024-03-04T15:54:31.242 INFO:teuthology.orchestra.run.smithi038.stdout:(24/32): ceph-common-19.0.0-1826.g9ca0b5de.el9. 6.9 MB/s | 20 MB 00:02 2024-03-04T15:54:31.267 INFO:teuthology.orchestra.run.smithi038.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 208 kB/s | 49 kB 00:00 2024-03-04T15:54:31.409 INFO:teuthology.orchestra.run.smithi038.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 404 kB/s | 67 kB 00:00 2024-03-04T15:54:31.476 INFO:teuthology.orchestra.run.smithi038.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 838 kB 00:00 2024-03-04T15:54:31.508 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-04T15:54:31.509 INFO:teuthology.orchestra.run.smithi038.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-04T15:54:31.643 INFO:teuthology.orchestra.run.smithi038.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-04T15:54:31.777 INFO:teuthology.orchestra.run.smithi038.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.3 MB/s | 1.6 MB 00:00 2024-03-04T15:54:31.893 INFO:teuthology.orchestra.run.smithi038.stdout:(31/32): librados2-19.0.0-1826.g9ca0b5de.el9.x8 8.8 MB/s | 3.4 MB 00:00 2024-03-04T15:54:31.960 INFO:teuthology.orchestra.run.smithi038.stdout:(32/32): librbd1-19.0.0-1826.g9ca0b5de.el9.x86_ 9.9 MB/s | 3.1 MB 00:00 2024-03-04T15:54:31.964 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:54:31.964 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-04T15:54:31.964 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:54:32.005 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:54:32.006 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:54:32.506 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:54:32.507 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:54:33.273 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-04T15:54:33.999 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:54:34.077 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T15:54:34.107 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-04T15:54:34.136 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-04T15:54:34.163 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T15:54:34.180 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T15:54:34.332 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T15:54:34.387 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T15:54:34.686 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-03-04T15:54:34.784 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-04T15:54:34.817 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:54:34.910 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:54:35.015 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 8/34 2024-03-04T15:54:35.061 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 9/34 2024-03-04T15:54:35.124 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 9/34 2024-03-04T15:54:35.153 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-04T15:54:35.196 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9. 11/34 2024-03-04T15:54:35.267 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 12/34 2024-03-04T15:54:35.296 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 13/34 2024-03-04T15:54:35.359 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 13/34 2024-03-04T15:54:35.387 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-04T15:54:35.499 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T15:54:35.530 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T15:54:35.570 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T15:54:35.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T15:54:35.640 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T15:54:35.673 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T15:54:36.004 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T15:54:36.028 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 22/34 2024-03-04T15:54:36.119 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 22/34 2024-03-04T15:54:36.149 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 23/34 2024-03-04T15:54:36.188 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T15:54:36.664 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T15:54:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 26/34 2024-03-04T15:54:36.752 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 26/34 2024-03-04T15:54:36.822 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 27/34 2024-03-04T15:54:36.849 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x8 28/34 2024-03-04T15:54:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:39.583 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-04T15:54:39.840 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:39.875 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 30/34 2024-03-04T15:54:39.965 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 30/34 2024-03-04T15:54:39.965 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T15:54:39.965 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T15:54:39.965 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:40.001 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:54:42.161 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 9.8 M 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrading: 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: librados2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.4 M 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.1 M 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-04T15:54:42.177 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 5.3 M 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 20 M 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 25 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 745 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 501 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-04T15:54:42.178 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2 x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 5.1 M 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 45 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 131 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 161 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 320 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 298 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 99 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-03-04T15:54:42.179 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout:Installing weak dependencies: 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout:Install 30 Packages 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout:Upgrade 2 Packages 2024-03-04T15:54:42.180 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:54:42.181 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 58 M 2024-03-04T15:54:42.181 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:54:43.659 INFO:teuthology.orchestra.run.smithi176.stdout:(1/32): ceph-common-19.0.0-1826.g9ca0b5de.el9.x 14 MB/s | 20 MB 00:01 2024-03-04T15:54:43.701 INFO:teuthology.orchestra.run.smithi176.stdout:(2/32): ceph-selinux-19.0.0-1826.g9ca0b5de.el9. 607 kB/s | 25 kB 00:00 2024-03-04T15:54:43.801 INFO:teuthology.orchestra.run.smithi176.stdout:(3/32): libcephfs2-19.0.0-1826.g9ca0b5de.el9.x8 7.3 MB/s | 745 kB 00:00 2024-03-04T15:54:43.877 INFO:teuthology.orchestra.run.smithi176.stdout:(4/32): libradosstriper1-19.0.0-1826.g9ca0b5de. 6.5 MB/s | 501 kB 00:00 2024-03-04T15:54:44.185 INFO:teuthology.orchestra.run.smithi176.stdout:(5/32): librgw2-19.0.0-1826.g9ca0b5de.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-04T15:54:44.236 INFO:teuthology.orchestra.run.smithi176.stdout:(6/32): python3-ceph-argparse-19.0.0-1826.g9ca0 907 kB/s | 45 kB 00:00 2024-03-04T15:54:44.303 INFO:teuthology.orchestra.run.smithi176.stdout:(7/32): ceph-base-19.0.0-1826.g9ca0b5de.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-04T15:54:44.328 INFO:teuthology.orchestra.run.smithi176.stdout:(8/32): python3-ceph-common-19.0.0-1826.g9ca0b5 1.4 MB/s | 131 kB 00:00 2024-03-04T15:54:44.370 INFO:teuthology.orchestra.run.smithi176.stdout:(9/32): python3-cephfs-19.0.0-1826.g9ca0b5de.el 2.4 MB/s | 161 kB 00:00 2024-03-04T15:54:44.395 INFO:teuthology.orchestra.run.smithi176.stdout:(10/32): python3-rados-19.0.0-1826.g9ca0b5de.el 4.7 MB/s | 320 kB 00:00 2024-03-04T15:54:44.437 INFO:teuthology.orchestra.run.smithi176.stdout:(11/32): python3-rbd-19.0.0-1826.g9ca0b5de.el9. 4.4 MB/s | 298 kB 00:00 2024-03-04T15:54:44.462 INFO:teuthology.orchestra.run.smithi176.stdout:(12/32): python3-rgw-19.0.0-1826.g9ca0b5de.el9. 1.5 MB/s | 99 kB 00:00 2024-03-04T15:54:44.612 INFO:teuthology.orchestra.run.smithi176.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 605 kB/s | 106 kB 00:00 2024-03-04T15:54:44.637 INFO:teuthology.orchestra.run.smithi176.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-04T15:54:44.654 INFO:teuthology.orchestra.run.smithi176.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-04T15:54:44.830 INFO:teuthology.orchestra.run.smithi176.stdout:(16/32): ceph-radosgw-19.0.0-1826.g9ca0b5de.el9 3.8 MB/s | 9.8 MB 00:02 2024-03-04T15:54:44.855 INFO:teuthology.orchestra.run.smithi176.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 661 kB 00:00 2024-03-04T15:54:44.880 INFO:teuthology.orchestra.run.smithi176.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.2 MB/s | 282 kB 00:00 2024-03-04T15:54:44.905 INFO:teuthology.orchestra.run.smithi176.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 556 kB/s | 41 kB 00:00 2024-03-04T15:54:44.931 INFO:teuthology.orchestra.run.smithi176.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 677 kB/s | 34 kB 00:00 2024-03-04T15:54:44.956 INFO:teuthology.orchestra.run.smithi176.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 215 kB 00:00 2024-03-04T15:54:44.981 INFO:teuthology.orchestra.run.smithi176.stdout:(22/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-03-04T15:54:45.165 INFO:teuthology.orchestra.run.smithi176.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 142 kB/s | 26 kB 00:00 2024-03-04T15:54:45.265 INFO:teuthology.orchestra.run.smithi176.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 926 kB/s | 309 kB 00:00 2024-03-04T15:54:45.290 INFO:teuthology.orchestra.run.smithi176.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 388 kB/s | 49 kB 00:00 2024-03-04T15:54:45.324 INFO:teuthology.orchestra.run.smithi176.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.2 MB/s | 67 kB 00:00 2024-03-04T15:54:45.399 INFO:teuthology.orchestra.run.smithi176.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-04T15:54:45.435 INFO:teuthology.orchestra.run.smithi176.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-04T15:54:45.458 INFO:teuthology.orchestra.run.smithi176.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-04T15:54:45.566 INFO:teuthology.orchestra.run.smithi176.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-04T15:54:45.717 INFO:teuthology.orchestra.run.smithi176.stdout:(31/32): librbd1-19.0.0-1826.g9ca0b5de.el9.x86_ 12 MB/s | 3.1 MB 00:00 2024-03-04T15:54:46.234 INFO:teuthology.orchestra.run.smithi176.stdout:(32/32): librados2-19.0.0-1826.g9ca0b5de.el9.x8 4.2 MB/s | 3.4 MB 00:00 2024-03-04T15:54:46.241 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:54:46.241 INFO:teuthology.orchestra.run.smithi176.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-04T15:54:46.242 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:54:46.279 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:54:46.279 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:54:46.749 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:54:46.749 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:54:48.199 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:54:48.289 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-04T15:54:48.330 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-04T15:54:48.372 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-04T15:54:48.395 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T15:54:48.411 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-04T15:54:48.577 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T15:54:48.644 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-04T15:54:48.985 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-04T15:54:49.032 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:54:49.133 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:54:49.222 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 8/34 2024-03-04T15:54:49.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 9/34 2024-03-04T15:54:49.321 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 9/34 2024-03-04T15:54:49.350 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-04T15:54:49.385 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9. 11/34 2024-03-04T15:54:49.455 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 12/34 2024-03-04T15:54:49.477 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 13/34 2024-03-04T15:54:49.539 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 13/34 2024-03-04T15:54:49.567 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-04T15:54:49.678 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-04T15:54:49.719 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-04T15:54:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-04T15:54:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-04T15:54:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-04T15:54:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-04T15:54:50.174 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-04T15:54:50.200 INFO:teuthology.orchestra.run.smithi176.stdout: Upgrading : librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 22/34 2024-03-04T15:54:50.204 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:54:50.204 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-03-04T15:54:50.204 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-03-04T15:54:50.204 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-03-04T15:54:50.205 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-03-04T15:54:50.205 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-03-04T15:54:50.205 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-03-04T15:54:50.205 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-03-04T15:54:50.205 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:50.290 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 22/34 2024-03-04T15:54:50.312 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 23/34 2024-03-04T15:54:50.366 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-04T15:54:50.826 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-04T15:54:50.861 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 26/34 2024-03-04T15:54:50.949 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 26/34 2024-03-04T15:54:51.016 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 27/34 2024-03-04T15:54:51.046 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x8 28/34 2024-03-04T15:54:51.141 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/34 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/34 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:54:51.167 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:51.169 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T15:54:51.278 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T15:54:51.278 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:54:52.411 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 3/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 4/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 6/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9. 8/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x8 9/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 10/34 2024-03-04T15:54:52.412 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 11/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 12/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 13/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T15:54:52.413 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T15:54:52.414 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T15:54:52.414 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T15:54:52.414 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T15:54:52.414 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T15:54:52.414 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T15:54:52.415 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:54:52.415 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T15:54:52.415 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/34 2024-03-04T15:54:53.244 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:53.284 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:53.475 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:54:53.475 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.476 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.477 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:53.478 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:54:53.829 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 29/34 2024-03-04T15:54:53.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-test 2024-03-04T15:54:53.862 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 30/34 2024-03-04T15:54:53.943 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 30/34 2024-03-04T15:54:53.943 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-04T15:54:53.943 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-04T15:54:53.943 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:54:53.980 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:54:54.430 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:29 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:54:54.616 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:54:54.616 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:54.616 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 47 M 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 163 k 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:54:54.617 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:54:54.618 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-03-04T15:54:54.618 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:54:54.618 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 48 M 2024-03-04T15:54:54.618 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 205 M 2024-03-04T15:54:54.618 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:54:54.829 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-04T15:54:54.871 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T15:54:54.930 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-03-04T15:54:54.980 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-04T15:54:55.022 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): libcephsqlite-19.0.0-1826.g9ca0b5de.el9. 406 kB/s | 163 kB 00:00 2024-03-04T15:54:57.313 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): ceph-test-19.0.0-1826.g9ca0b5de.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-04T15:54:57.317 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:54:57.317 INFO:teuthology.orchestra.run.smithi038.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-04T15:54:57.317 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:54:57.339 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:54:57.339 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:54:57.757 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:54:57.757 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:54:58.345 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:54:58.412 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T15:54:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T15:54:58.499 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T15:54:58.548 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T15:54:58.575 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/6 2024-03-04T15:55:03.315 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/6 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /sys 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /proc 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /mnt 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /var/tmp 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /home 2024-03-04T15:55:03.316 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /root 2024-03-04T15:55:03.317 INFO:teuthology.orchestra.run.smithi176.stdout:skipping the directory /tmp 2024-03-04T15:55:03.317 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:03.394 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 6/6 2024-03-04T15:55:04.240 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/34 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/34 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T15:55:04.610 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-04T15:55:04.611 INFO:teuthology.orchestra.run.smithi176.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 3/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 4/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_6 6/34 2024-03-04T15:55:05.439 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 7/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9. 8/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x8 9/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 10/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 11/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 12/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 13/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-04T15:55:05.440 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 31/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-04T15:55:05.441 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/34 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 6/6 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/6 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/6 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T15:55:05.909 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-16.el9.x86_64 2024-03-04T15:55:06.313 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T15:55:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T15:55:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T15:55:06.314 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:06.314 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:06.452 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-04T15:55:06.452 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:06.452 INFO:teuthology.orchestra.run.smithi176.stdout:Upgraded: 2024-03-04T15:55:06.452 INFO:teuthology.orchestra.run.smithi176.stdout: librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-radosgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-selinux-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-04T15:55:06.453 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: librgw2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-argparse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.454 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: re2-1:20211101-3.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:06.455 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:06.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph 2024-03-04T15:55:06.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-test 2024-03-04T15:55:07.212 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:42 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:07.332 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:28 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:07.401 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 6.6 k 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 2.1 M 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 1.5 M 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 248 k 2024-03-04T15:55:07.404 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 4.7 M 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 17 M 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T15:55:07.405 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T15:55:07.406 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout:========================================================================================= 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout:Install 36 Packages 2024-03-04T15:55:07.407 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:07.408 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 30 M 2024-03-04T15:55:07.408 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 107 M 2024-03-04T15:55:07.408 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:07.497 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:07.498 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:07.498 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:07.498 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:07.498 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 47 M 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 163 k 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-04T15:55:07.499 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:07.500 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 48 M 2024-03-04T15:55:07.500 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 205 M 2024-03-04T15:55:07.500 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:07.705 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-04T15:55:07.747 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-04T15:55:07.797 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-03-04T15:55:07.856 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-04T15:55:07.881 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): libcephsqlite-19.0.0-1826.g9ca0b5de.el9. 429 kB/s | 163 kB 00:00 2024-03-04T15:55:08.275 INFO:teuthology.orchestra.run.smithi038.stdout:(1/36): ceph-19.0.0-1826.g9ca0b5de.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-04T15:55:08.683 INFO:teuthology.orchestra.run.smithi038.stdout:(2/36): ceph-mds-19.0.0-1826.g9ca0b5de.el9.x86_ 3.7 MB/s | 2.1 MB 00:00 2024-03-04T15:55:09.742 INFO:teuthology.orchestra.run.smithi038.stdout:(3/36): ceph-mgr-19.0.0-1826.g9ca0b5de.el9.x86_ 937 kB/s | 1.5 MB 00:01 2024-03-04T15:55:10.000 INFO:teuthology.orchestra.run.smithi038.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1826.g9ca0 960 kB/s | 248 kB 00:00 2024-03-04T15:55:10.150 INFO:teuthology.orchestra.run.smithi038.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-04T15:55:10.209 INFO:teuthology.orchestra.run.smithi038.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-04T15:55:10.251 INFO:teuthology.orchestra.run.smithi038.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-04T15:55:10.293 INFO:teuthology.orchestra.run.smithi038.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-04T15:55:10.352 INFO:teuthology.orchestra.run.smithi038.stdout:(9/36): ceph-mon-19.0.0-1826.g9ca0b5de.el9.x86_ 2.3 MB/s | 4.7 MB 00:02 2024-03-04T15:55:10.377 INFO:teuthology.orchestra.run.smithi038.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 461 kB/s | 39 kB 00:00 2024-03-04T15:55:10.411 INFO:teuthology.orchestra.run.smithi038.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-04T15:55:10.478 INFO:teuthology.orchestra.run.smithi038.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-04T15:55:10.503 INFO:teuthology.orchestra.run.smithi038.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-04T15:55:10.528 INFO:teuthology.orchestra.run.smithi038.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-04T15:55:10.545 INFO:teuthology.orchestra.run.smithi038.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.7 MB/s | 113 kB 00:00 2024-03-04T15:55:10.579 INFO:teuthology.orchestra.run.smithi038.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-04T15:55:10.623 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): ceph-test-19.0.0-1826.g9ca0b5de.el9.x86_ 15 MB/s | 47 MB 00:03 2024-03-04T15:55:10.626 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:10.626 INFO:teuthology.orchestra.run.smithi176.stdout:Total 15 MB/s | 48 MB 00:03 2024-03-04T15:55:10.626 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:10.646 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:10.646 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:10.662 INFO:teuthology.orchestra.run.smithi038.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-03-04T15:55:11.013 INFO:teuthology.orchestra.run.smithi038.stdout:(18/36): ceph-osd-19.0.0-1826.g9ca0b5de.el9.x86 7.2 MB/s | 17 MB 00:02 2024-03-04T15:55:11.038 INFO:teuthology.orchestra.run.smithi038.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-04T15:55:11.041 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:11.041 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:11.146 INFO:teuthology.orchestra.run.smithi038.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-03-04T15:55:11.305 INFO:teuthology.orchestra.run.smithi038.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2024-03-04T15:55:11.330 INFO:teuthology.orchestra.run.smithi038.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 542 kB/s | 172 kB 00:00 2024-03-04T15:55:11.381 INFO:teuthology.orchestra.run.smithi038.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-03-04T15:55:11.497 INFO:teuthology.orchestra.run.smithi038.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 92 kB/s | 18 kB 00:00 2024-03-04T15:55:11.523 INFO:teuthology.orchestra.run.smithi038.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 121 kB/s | 23 kB 00:00 2024-03-04T15:55:11.581 INFO:teuthology.orchestra.run.smithi038.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 97 kB/s | 19 kB 00:00 2024-03-04T15:55:11.616 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:11.679 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-03-04T15:55:11.681 INFO:teuthology.orchestra.run.smithi038.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 107 kB/s | 20 kB 00:00 2024-03-04T15:55:11.729 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-03-04T15:55:11.757 INFO:teuthology.orchestra.run.smithi038.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 198 kB/s | 46 kB 00:00 2024-03-04T15:55:11.774 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-03-04T15:55:11.816 INFO:teuthology.orchestra.run.smithi038.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 337 kB/s | 79 kB 00:00 2024-03-04T15:55:11.822 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-04T15:55:11.851 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/6 2024-03-04T15:55:11.949 INFO:teuthology.orchestra.run.smithi038.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 86 kB/s | 16 kB 00:00 2024-03-04T15:55:12.041 INFO:teuthology.orchestra.run.smithi038.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 758 kB/s | 272 kB 00:00 2024-03-04T15:55:12.083 INFO:teuthology.orchestra.run.smithi038.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 337 kB/s | 90 kB 00:00 2024-03-04T15:55:12.158 INFO:teuthology.orchestra.run.smithi038.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 171 kB/s | 36 kB 00:00 2024-03-04T15:55:12.350 INFO:teuthology.orchestra.run.smithi038.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 104 kB/s | 20 kB 00:00 2024-03-04T15:55:12.375 INFO:teuthology.orchestra.run.smithi038.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 688 kB/s | 230 kB 00:00 2024-03-04T15:55:12.442 INFO:teuthology.orchestra.run.smithi038.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.2 MB/s | 427 kB 00:00 2024-03-04T15:55:12.448 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:12.449 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.9 MB/s | 30 MB 00:05 2024-03-04T15:55:12.469 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:12.515 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:12.515 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:12.942 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:12.942 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:14.283 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:14.353 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-04T15:55:14.396 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-04T15:55:14.422 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-04T15:55:14.490 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-04T15:55:14.530 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-04T15:55:14.562 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-04T15:55:14.660 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-04T15:55:14.745 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-04T15:55:14.791 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-04T15:55:14.865 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-04T15:55:14.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-04T15:55:14.954 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-04T15:55:15.013 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-04T15:55:15.088 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-04T15:55:15.297 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-04T15:55:15.342 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-04T15:55:15.380 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-04T15:55:15.415 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-04T15:55:15.436 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-04T15:55:15.515 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-04T15:55:15.547 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T15:55:15.580 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-04T15:55:15.621 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-04T15:55:15.664 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-04T15:55:15.698 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-04T15:55:15.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-04T15:55:15.829 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-04T15:55:15.906 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-04T15:55:16.055 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-04T15:55:16.187 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T15:55:16.298 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/6 2024-03-04T15:55:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9. 31/36 2024-03-04T15:55:16.376 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 6/6 2024-03-04T15:55:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/36 2024-03-04T15:55:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/36 2024-03-04T15:55:16.409 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:16.409 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T15:55:16.409 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:55:16.409 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:55:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:17.917 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/36 2024-03-04T15:55:17.943 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/36 2024-03-04T15:55:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T15:55:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:55:17.943 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:55:17.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:18.401 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 34/36 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 34/36 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:55:18.424 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:18.636 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 35/36 2024-03-04T15:55:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 35/36 2024-03-04T15:55:18.658 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:18.658 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T15:55:18.658 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:55:18.659 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:55:18.659 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:18.691 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 6/6 2024-03-04T15:55:18.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/6 2024-03-04T15:55:18.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/6 2024-03-04T15:55:18.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-03-04T15:55:18.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-03-04T15:55:18.692 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-03-04T15:55:18.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 36/36 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-test-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: jq-1.6-16.el9.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: libcephsqlite-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-04T15:55:19.067 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:19.068 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:19.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph 2024-03-04T15:55:19.930 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:40 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:20.108 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:20.110 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 6.6 k 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 2.1 M 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 1.5 M 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 248 k 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 4.7 M 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 17 M 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-04T15:55:20.111 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-04T15:55:20.112 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-03-04T15:55:20.113 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout:========================================================================================= 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout:Install 36 Packages 2024-03-04T15:55:20.114 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:20.115 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 30 M 2024-03-04T15:55:20.115 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 107 M 2024-03-04T15:55:20.115 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 36/36 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/36 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/36 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 3/36 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 4/36 2024-03-04T15:55:20.477 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9. 6/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-04T15:55:20.478 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-04T15:55:20.479 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-04T15:55:20.480 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-04T15:55:20.481 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-04T15:55:20.481 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-04T15:55:20.575 INFO:teuthology.orchestra.run.smithi176.stdout:(1/36): ceph-19.0.0-1826.g9ca0b5de.el9.x86_64.r 32 kB/s | 6.6 kB 00:00 2024-03-04T15:55:20.700 INFO:teuthology.orchestra.run.smithi176.stdout:(2/36): ceph-mgr-19.0.0-1826.g9ca0b5de.el9.x86_ 4.5 MB/s | 1.5 MB 00:00 2024-03-04T15:55:21.134 INFO:teuthology.orchestra.run.smithi176.stdout:(3/36): ceph-mds-19.0.0-1826.g9ca0b5de.el9.x86_ 2.8 MB/s | 2.1 MB 00:00 2024-03-04T15:55:21.242 INFO:teuthology.orchestra.run.smithi176.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1826.g9ca0 2.2 MB/s | 248 kB 00:00 2024-03-04T15:55:21.426 INFO:teuthology.orchestra.run.smithi176.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:21.524 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T15:55:21.525 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T15:55:21.526 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:21.527 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:21.634 INFO:teuthology.orchestra.run.smithi176.stdout:(6/36): ceph-osd-19.0.0-1826.g9ca0b5de.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-04T15:55:21.660 INFO:teuthology.orchestra.run.smithi176.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 617 kB/s | 144 kB 00:00 2024-03-04T15:55:21.685 INFO:teuthology.orchestra.run.smithi176.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-04T15:55:21.718 INFO:teuthology.orchestra.run.smithi176.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 807 kB/s | 47 kB 00:00 2024-03-04T15:55:21.752 INFO:teuthology.orchestra.run.smithi176.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 576 kB/s | 39 kB 00:00 2024-03-04T15:55:21.827 INFO:teuthology.orchestra.run.smithi176.stdout:(11/36): ceph-mon-19.0.0-1826.g9ca0b5de.el9.x86 3.7 MB/s | 4.7 MB 00:01 2024-03-04T15:55:21.853 INFO:teuthology.orchestra.run.smithi176.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 241 kB 00:00 2024-03-04T15:55:21.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-base 2024-03-04T15:55:21.995 INFO:teuthology.orchestra.run.smithi176.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 617 kB/s | 103 kB 00:00 2024-03-04T15:55:22.029 INFO:teuthology.orchestra.run.smithi176.stdout:(14/36): python3-cryptography-36.0.1-4.el9.x86_ 4.1 MB/s | 1.1 MB 00:00 2024-03-04T15:55:22.054 INFO:teuthology.orchestra.run.smithi176.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 619 kB/s | 124 kB 00:00 2024-03-04T15:55:22.079 INFO:teuthology.orchestra.run.smithi176.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 113 kB 00:00 2024-03-04T15:55:22.104 INFO:teuthology.orchestra.run.smithi176.stdout:(17/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-04T15:55:22.196 INFO:teuthology.orchestra.run.smithi176.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-04T15:55:22.437 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:22.496 INFO:teuthology.orchestra.run.smithi176.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 118 kB/s | 43 kB 00:00 2024-03-04T15:55:22.555 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:55:22.611 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:22.612 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:55:22.612 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:22.613 INFO:teuthology.orchestra.run.smithi176.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 283 kB/s | 151 kB 00:00 2024-03-04T15:55:22.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install cephadm 2024-03-04T15:55:22.705 INFO:teuthology.orchestra.run.smithi176.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-04T15:55:22.731 INFO:teuthology.orchestra.run.smithi176.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 353 kB/s | 172 kB 00:00 2024-03-04T15:55:22.797 INFO:teuthology.orchestra.run.smithi176.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-04T15:55:22.823 INFO:teuthology.orchestra.run.smithi176.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-03-04T15:55:22.848 INFO:teuthology.orchestra.run.smithi176.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-04T15:55:22.890 INFO:teuthology.orchestra.run.smithi176.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 210 kB/s | 19 kB 00:00 2024-03-04T15:55:22.915 INFO:teuthology.orchestra.run.smithi176.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-04T15:55:22.940 INFO:teuthology.orchestra.run.smithi176.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-04T15:55:22.999 INFO:teuthology.orchestra.run.smithi176.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-04T15:55:23.049 INFO:teuthology.orchestra.run.smithi176.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-04T15:55:23.074 INFO:teuthology.orchestra.run.smithi176.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-04T15:55:23.117 INFO:teuthology.orchestra.run.smithi176.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 767 kB/s | 90 kB 00:00 2024-03-04T15:55:23.150 INFO:teuthology.orchestra.run.smithi176.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-04T15:55:23.192 INFO:teuthology.orchestra.run.smithi176.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-04T15:55:23.242 INFO:teuthology.orchestra.run.smithi176.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-04T15:55:23.252 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:58 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:23.342 INFO:teuthology.orchestra.run.smithi176.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2024-03-04T15:55:23.349 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:23.350 INFO:teuthology.orchestra.run.smithi176.stdout:Total 9.2 MB/s | 30 MB 00:03 2024-03-04T15:55:23.367 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:23.410 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:23.410 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:23.439 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:23.440 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:23.440 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:23.440 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 748 k 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 748 k 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 755 k 2024-03-04T15:55:23.441 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:23.710 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm-19.0.0-1826.g9ca0b5de.el9.noarch.rpm 2.7 MB/s | 748 kB 00:00 2024-03-04T15:55:23.710 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:23.710 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.7 MB/s | 748 kB 00:00 2024-03-04T15:55:23.711 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:23.716 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:23.716 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:23.722 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:23.722 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:23.799 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:23.819 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:23.820 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:24.210 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:24.292 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:24.702 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:25.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:25.014 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:25.014 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:25.014 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:25.014 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:25.015 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:25.176 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:25.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T15:55:25.245 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-04T15:55:25.282 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-04T15:55:25.314 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-04T15:55:25.386 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-04T15:55:25.448 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-04T15:55:25.481 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-04T15:55:25.572 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-04T15:55:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-04T15:55:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-04T15:55:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-04T15:55:25.816 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:00 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-04T15:55:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-04T15:55:25.921 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-04T15:55:25.979 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-04T15:55:25.990 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:25.990 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 145 k 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 145 k 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 436 k 2024-03-04T15:55:25.991 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:26.165 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-04T15:55:26.191 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-immutable-object-cache-19.0.0-1826.g9ca0b5 728 kB/s | 145 kB 00:00 2024-03-04T15:55:26.191 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:26.192 INFO:teuthology.orchestra.run.smithi038.stdout:Total 722 kB/s | 145 kB 00:00 2024-03-04T15:55:26.192 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:26.200 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:26.200 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:26.210 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-04T15:55:26.248 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-04T15:55:26.254 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:26.254 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:26.283 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-04T15:55:26.313 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-04T15:55:26.389 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-04T15:55:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:26.423 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T15:55:26.457 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-04T15:55:26.484 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:26.497 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-04T15:55:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:26.508 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:26.508 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T15:55:26.508 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:26.532 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-04T15:55:26.567 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-04T15:55:26.655 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-04T15:55:26.689 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-04T15:55:26.762 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-04T15:55:26.898 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-04T15:55:27.031 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T15:55:27.199 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9. 31/36 2024-03-04T15:55:27.227 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/36 2024-03-04T15:55:27.252 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 32/36 2024-03-04T15:55:27.252 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:27.252 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-04T15:55:27.252 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:55:27.252 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-04T15:55:27.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:27.495 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:27.496 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:27.496 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:27.496 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:27.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr 2024-03-04T15:55:28.318 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:03 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:28.438 INFO:teuthology.orchestra.run.smithi038.stdout:Package ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:55:28.493 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:28.495 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:55:28.495 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:28.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T15:55:28.687 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/36 2024-03-04T15:55:28.712 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 33/36 2024-03-04T15:55:28.712 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:28.712 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-04T15:55:28.713 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:55:28.713 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-04T15:55:28.713 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:29.115 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:04 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:29.154 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 34/36 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 34/36 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-04T15:55:29.177 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:29.291 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:29.292 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-03-04T15:55:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:29.292 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-03-04T15:55:29.292 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:29.292 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 1.8 M 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 31 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 16 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:29.293 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:====================================================================================== 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 6.9 M 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-03-04T15:55:29.294 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:29.379 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 35/36 2024-03-04T15:55:29.402 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 35/36 2024-03-04T15:55:29.402 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:29.402 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-04T15:55:29.402 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:55:29.403 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-04T15:55:29.403 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:29.478 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 36/36 2024-03-04T15:55:30.062 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1826.g9ca 83 kB/s | 16 kB 00:00 2024-03-04T15:55:30.237 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1826.g9c 84 kB/s | 31 kB 00:00 2024-03-04T15:55:30.496 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-03-04T15:55:30.521 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 838 kB/s | 238 kB 00:00 2024-03-04T15:55:30.680 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-03-04T15:55:30.772 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-04T15:55:30.813 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1826.g9ca0b5d 1.9 MB/s | 1.8 MB 00:00 2024-03-04T15:55:30.847 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-04T15:55:30.889 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 742 kB/s | 31 kB 00:00 2024-03-04T15:55:30.973 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-04T15:55:30.998 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 786 kB/s | 144 kB 00:00 2024-03-04T15:55:31.183 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 36/36 2024-03-04T15:55:31.183 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/36 2024-03-04T15:55:31.183 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 3/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 4/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 5/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9. 6/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-04T15:55:31.184 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-04T15:55:31.185 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-04T15:55:31.186 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-04T15:55:31.189 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-04T15:55:31.194 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:31.194 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-04T15:55:31.214 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:31.234 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:31.234 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:31.420 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:31.421 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:32.075 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:32.149 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-04T15:55:32.190 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-04T15:55:32.190 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mds-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mon-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-osd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: lua-5.4.4-4.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-04T15:55:32.191 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ply-3.11-14.el9.noarch 2024-03-04T15:55:32.192 INFO:teuthology.orchestra.run.smithi176.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:32.193 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-04T15:55:32.248 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-04T15:55:32.277 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-04T15:55:32.371 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-04T15:55:32.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-04T15:55:32.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-base 2024-03-04T15:55:32.675 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-04T15:55:32.724 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-04T15:55:32.781 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-04T15:55:32.817 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9 10/12 2024-03-04T15:55:33.067 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el 11/12 2024-03-04T15:55:33.071 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:54 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:33.100 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 12/12 2024-03-04T15:55:33.185 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-base-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:55:33.238 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:33.239 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:55:33.239 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:33.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install cephadm 2024-03-04T15:55:33.865 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 12/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el 1/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 2/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9 3/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-04T15:55:33.866 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-04T15:55:33.867 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:54 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:34.033 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 748 k 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:34.034 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 748 k 2024-03-04T15:55:34.035 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 755 k 2024-03-04T15:55:34.035 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:34.245 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:34.246 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:34.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T15:55:34.736 INFO:teuthology.orchestra.run.smithi176.stdout:cephadm-19.0.0-1826.g9ca0b5de.el9.noarch.rpm 1.0 MB/s | 748 kB 00:00 2024-03-04T15:55:34.737 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:34.737 INFO:teuthology.orchestra.run.smithi176.stdout:Total 1.0 MB/s | 748 kB 00:00 2024-03-04T15:55:34.737 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:34.743 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:34.743 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:34.749 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:34.749 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:34.820 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:35.047 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:10 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:35.221 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 7.4 M 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T15:55:35.223 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout:Install 12 Packages 2024-03-04T15:55:35.224 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:35.225 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 37 M 2024-03-04T15:55:35.225 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 209 M 2024-03-04T15:55:35.225 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:35.263 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:35.377 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:35.546 INFO:teuthology.orchestra.run.smithi038.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 175 kB/s | 32 kB 00:00 2024-03-04T15:55:35.875 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:35.876 INFO:teuthology.orchestra.run.smithi038.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2024-03-04T15:55:35.876 INFO:teuthology.orchestra.run.smithi038.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 590 kB/s | 34 kB 00:00 2024-03-04T15:55:35.876 INFO:teuthology.orchestra.run.smithi038.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.0 MB/s | 3.0 MB 00:00 2024-03-04T15:55:35.964 INFO:teuthology.orchestra.run.smithi038.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-03-04T15:55:36.031 INFO:teuthology.orchestra.run.smithi038.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-04T15:55:36.056 INFO:teuthology.orchestra.run.smithi038.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/1 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout: cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:36.060 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:36.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-immutable-object-cache 2024-03-04T15:55:36.457 INFO:teuthology.orchestra.run.smithi038.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-04T15:55:36.490 INFO:teuthology.orchestra.run.smithi038.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 802 kB 00:00 2024-03-04T15:55:36.515 INFO:teuthology.orchestra.run.smithi038.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 194 kB 00:00 2024-03-04T15:55:36.607 INFO:teuthology.orchestra.run.smithi038.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.0 MB/s | 7.4 MB 00:01 2024-03-04T15:55:36.852 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:00:57 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:37.021 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:37.022 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:37.022 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-04T15:55:37.022 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 145 k 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 145 k 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 436 k 2024-03-04T15:55:37.023 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:37.124 INFO:teuthology.orchestra.run.smithi038.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 387 kB/s | 245 kB 00:00 2024-03-04T15:55:37.127 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:37.127 INFO:teuthology.orchestra.run.smithi038.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-04T15:55:37.142 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:37.169 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:37.169 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:37.384 INFO:teuthology.orchestra.run.smithi176.stdout:ceph-immutable-object-cache-19.0.0-1826.g9ca0b5 402 kB/s | 145 kB 00:00 2024-03-04T15:55:37.384 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:37.385 INFO:teuthology.orchestra.run.smithi176.stdout:Total 400 kB/s | 145 kB 00:00 2024-03-04T15:55:37.385 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:37.392 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:37.392 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:37.445 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:37.445 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:37.549 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:37.584 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:37.651 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:37.675 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:37.676 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:37.676 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-04T15:55:37.676 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T15:55:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T15:55:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:55:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de. 1/1 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-immutable-object-cache-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:38.682 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:55:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:55:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T15:55:38.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr 2024-03-04T15:55:39.488 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:00 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:39.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T15:55:39.602 INFO:teuthology.orchestra.run.smithi176.stdout:Package ceph-mgr-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:55:39.613 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T15:55:39.654 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:39.655 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:55:39.655 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:39.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-dashboard 2024-03-04T15:55:40.282 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:01 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:40.451 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 1.8 M 2024-03-04T15:55:40.452 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 31 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 16 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-04T15:55:40.453 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:====================================================================================== 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.9 M 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-04T15:55:40.454 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:40.930 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T15:55:41.017 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1826.g9ca 48 kB/s | 16 kB 00:00 2024-03-04T15:55:41.021 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1826.g9c 88 kB/s | 31 kB 00:00 2024-03-04T15:55:41.207 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-04T15:55:41.238 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.3 MB/s | 1.0 MB 00:00 2024-03-04T15:55:41.285 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1826.g9ca0b5d 2.9 MB/s | 1.8 MB 00:00 2024-03-04T15:55:41.506 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-03-04T15:55:41.698 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 49 kB/s | 19 kB 00:00 2024-03-04T15:55:41.973 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 746 kB/s | 548 kB 00:00 2024-03-04T15:55:41.991 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T15:55:42.030 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 12/12 2024-03-04T15:55:42.031 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 434 kB/s | 144 kB 00:00 2024-03-04T15:55:42.082 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-04T15:55:42.232 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 942 kB/s | 188 kB 00:00 2024-03-04T15:55:42.424 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-04T15:55:42.429 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:42.430 INFO:teuthology.orchestra.run.smithi176.stdout:Total 3.5 MB/s | 6.9 MB 00:01 2024-03-04T15:55:42.447 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:42.464 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:42.464 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:42.638 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:42.638 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:43.153 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:43.232 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-04T15:55:43.267 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-04T15:55:43.315 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-04T15:55:43.345 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-04T15:55:43.454 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-04T15:55:43.606 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-04T15:55:43.697 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-04T15:55:43.757 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-04T15:55:43.786 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-04T15:55:43.817 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9 10/12 2024-03-04T15:55:44.054 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el 11/12 2024-03-04T15:55:44.084 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 12/12 2024-03-04T15:55:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 12/12 2024-03-04T15:55:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 1/12 2024-03-04T15:55:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T15:55:44.607 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T15:55:44.608 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 12/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el 1/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noa 2/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9 3/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-04T15:55:44.891 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-04T15:55:44.892 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-04T15:55:44.892 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-04T15:55:44.892 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-04T15:55:44.892 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T15:55:45.025 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:45.026 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-grafana-dashboards-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-prometheus-alerts-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-04T15:55:45.253 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:45.254 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:45.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-rook 2024-03-04T15:55:45.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-04T15:55:45.888 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:20 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:46.033 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:07 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:46.062 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:46.063 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:46.063 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:46.063 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:46.063 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:46.063 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 49 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T15:55:46.064 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout:Install 13 Packages 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:46.065 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 2.0 M 2024-03-04T15:55:46.066 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 26 M 2024-03-04T15:55:46.066 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:46.201 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:46.202 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-04T15:55:46.202 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:46.202 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-04T15:55:46.202 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 7.4 M 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-04T15:55:46.203 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:46.204 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:46.204 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================================ 2024-03-04T15:55:46.204 INFO:teuthology.orchestra.run.smithi176.stdout:Install 12 Packages 2024-03-04T15:55:46.204 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:46.204 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 37 M 2024-03-04T15:55:46.205 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 209 M 2024-03-04T15:55:46.205 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:46.356 INFO:teuthology.orchestra.run.smithi038.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 24 kB 00:00 2024-03-04T15:55:46.381 INFO:teuthology.orchestra.run.smithi038.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-04T15:55:46.448 INFO:teuthology.orchestra.run.smithi038.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-04T15:55:46.474 INFO:teuthology.orchestra.run.smithi038.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-04T15:55:46.499 INFO:teuthology.orchestra.run.smithi038.stdout:(5/13): ceph-mgr-rook-19.0.0-1826.g9ca0b5de.el9 177 kB/s | 49 kB 00:00 2024-03-04T15:55:46.524 INFO:teuthology.orchestra.run.smithi038.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-04T15:55:46.541 INFO:teuthology.orchestra.run.smithi038.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 638 kB/s | 43 kB 00:00 2024-03-04T15:55:46.647 INFO:teuthology.orchestra.run.smithi176.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 189 kB/s | 32 kB 00:00 2024-03-04T15:55:46.680 INFO:teuthology.orchestra.run.smithi176.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-04T15:55:46.714 INFO:teuthology.orchestra.run.smithi176.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-04T15:55:46.741 INFO:teuthology.orchestra.run.smithi038.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-03-04T15:55:46.758 INFO:teuthology.orchestra.run.smithi038.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 69 kB/s | 15 kB 00:00 2024-03-04T15:55:46.764 INFO:teuthology.orchestra.run.smithi176.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-04T15:55:46.842 INFO:teuthology.orchestra.run.smithi038.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 711 kB/s | 59 kB 00:00 2024-03-04T15:55:46.892 INFO:teuthology.orchestra.run.smithi038.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 578 kB/s | 203 kB 00:00 2024-03-04T15:55:46.948 INFO:teuthology.orchestra.run.smithi176.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-04T15:55:46.959 INFO:teuthology.orchestra.run.smithi038.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-03-04T15:55:47.015 INFO:teuthology.orchestra.run.smithi176.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-04T15:55:47.040 INFO:teuthology.orchestra.run.smithi176.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-04T15:55:47.051 INFO:teuthology.orchestra.run.smithi038.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-04T15:55:47.054 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:47.055 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-04T15:55:47.055 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:47.069 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:47.069 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:47.171 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:47.172 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:47.182 INFO:teuthology.orchestra.run.smithi176.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-03-04T15:55:47.374 INFO:teuthology.orchestra.run.smithi176.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-04T15:55:47.399 INFO:teuthology.orchestra.run.smithi176.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 905 kB/s | 194 kB 00:00 2024-03-04T15:55:47.708 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:47.879 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T15:55:47.928 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T15:55:47.984 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T15:55:48.028 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T15:55:48.070 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T15:55:48.074 INFO:teuthology.orchestra.run.smithi176.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-03-04T15:55:48.166 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T15:55:48.283 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T15:55:48.338 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T15:55:48.383 INFO:teuthology.orchestra.run.smithi176.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 243 kB/s | 245 kB 00:01 2024-03-04T15:55:48.386 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:48.386 INFO:teuthology.orchestra.run.smithi176.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-04T15:55:48.399 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:48.424 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:48.425 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:48.773 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:48.773 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:49.102 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T15:55:49.143 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T15:55:49.177 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T15:55:49.226 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T15:55:49.256 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 13/13 2024-03-04T15:55:49.484 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:49.556 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-04T15:55:49.587 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-04T15:55:49.684 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:55:49.732 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-04T15:55:49.910 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:55:49.947 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:55:50.013 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 13/13 2024-03-04T15:55:50.013 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/13 2024-03-04T15:55:50.013 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T15:55:50.014 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T15:55:50.023 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T15:55:50.396 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:50.397 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:50.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T15:55:50.680 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-04T15:55:50.810 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-04T15:55:51.210 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:26 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:51.387 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:51.387 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:51.387 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 138 k 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:======================================================================================== 2024-03-04T15:55:51.388 INFO:teuthology.orchestra.run.smithi038.stdout:Install 6 Packages 2024-03-04T15:55:51.389 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:51.389 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 6.8 M 2024-03-04T15:55:51.389 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 33 M 2024-03-04T15:55:51.389 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:51.680 INFO:teuthology.orchestra.run.smithi038.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 951 kB/s | 227 kB 00:00 2024-03-04T15:55:51.856 INFO:teuthology.orchestra.run.smithi038.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-04T15:55:51.881 INFO:teuthology.orchestra.run.smithi038.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1826.g9ca0b5de.e 315 kB/s | 138 kB 00:00 2024-03-04T15:55:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 612 kB/s | 51 kB 00:00 2024-03-04T15:55:51.990 INFO:teuthology.orchestra.run.smithi038.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 431 kB/s | 58 kB 00:00 2024-03-04T15:55:52.042 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-04T15:55:52.140 INFO:teuthology.orchestra.run.smithi038.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-04T15:55:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:Total 9.0 MB/s | 6.8 MB 00:00 2024-03-04T15:55:52.142 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:52.152 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:52.152 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:52.246 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:52.246 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:52.560 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:52.736 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T15:55:52.792 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T15:55:53.061 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-04T15:55:53.089 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 12/12 2024-03-04T15:55:53.348 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T15:55:53.437 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T15:55:53.499 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T15:55:53.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 6/6 2024-03-04T15:55:54.343 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 6/6 2024-03-04T15:55:54.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/6 2024-03-04T15:55:54.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T15:55:54.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T15:55:54.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T15:55:54.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T15:55:54.680 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T15:55:54.681 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T15:55:54.681 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:54.681 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:54.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-fuse 2024-03-04T15:55:55.502 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 12/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b 1/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-04T15:55:55.597 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-04T15:55:55.598 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-04T15:55:55.598 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-04T15:55:55.598 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-04T15:55:55.598 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-04T15:55:55.678 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 888 k 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:55.680 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 966 k 2024-03-04T15:55:55.681 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 2.8 M 2024-03-04T15:55:55.682 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:55.850 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 465 kB/s | 79 kB 00:00 2024-03-04T15:55:56.041 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-fuse-19.0.0-1826.g9ca0b5de.el9.x86_ 2.4 MB/s | 888 kB 00:00 2024-03-04T15:55:56.043 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:56.043 INFO:teuthology.orchestra.run.smithi038.stdout:Total 2.6 MB/s | 966 kB 00:00 2024-03-04T15:55:56.043 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:56.058 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-04T15:55:56.059 INFO:teuthology.orchestra.run.smithi176.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-04T15:55:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-04T15:55:56.060 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:56.060 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:55:56.064 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:56.064 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:56.143 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:56.143 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:56.331 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:56.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-rook 2024-03-04T15:55:56.464 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T15:55:56.537 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/2 2024-03-04T15:55:56.990 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:17 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:55:57.159 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:55:57.160 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:55:57.160 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:55:57.160 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:55:57.160 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:55:57.160 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 49 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-04T15:55:57.161 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout:Install 13 Packages 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:55:57.162 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 2.0 M 2024-03-04T15:55:57.163 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 26 M 2024-03-04T15:55:57.163 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:55:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/2 2024-03-04T15:55:57.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/2 2024-03-04T15:55:57.389 INFO:teuthology.orchestra.run.smithi176.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 176 kB/s | 24 kB 00:00 2024-03-04T15:55:57.406 INFO:teuthology.orchestra.run.smithi176.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-04T15:55:57.490 INFO:teuthology.orchestra.run.smithi176.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-04T15:55:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T15:55:57.502 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:57.502 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:55:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T15:55:57.503 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:57.503 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:55:57.515 INFO:teuthology.orchestra.run.smithi176.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-04T15:55:57.532 INFO:teuthology.orchestra.run.smithi176.stdout:(5/13): ceph-mgr-rook-19.0.0-1826.g9ca0b5de.el9 176 kB/s | 49 kB 00:00 2024-03-04T15:55:57.557 INFO:teuthology.orchestra.run.smithi176.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.1 MB/s | 211 kB 00:00 2024-03-04T15:55:57.574 INFO:teuthology.orchestra.run.smithi176.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 729 kB/s | 43 kB 00:00 2024-03-04T15:55:57.708 INFO:teuthology.orchestra.run.smithi176.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-03-04T15:55:57.725 INFO:teuthology.orchestra.run.smithi176.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-04T15:55:57.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-volume 2024-03-04T15:55:57.775 INFO:teuthology.orchestra.run.smithi176.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 1.0 MB/s | 203 kB 00:00 2024-03-04T15:55:57.792 INFO:teuthology.orchestra.run.smithi176.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 882 kB/s | 59 kB 00:00 2024-03-04T15:55:57.825 INFO:teuthology.orchestra.run.smithi176.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-04T15:55:57.875 INFO:teuthology.orchestra.run.smithi176.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-03-04T15:55:57.880 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:57.880 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-04T15:55:57.880 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:55:57.893 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:55:57.893 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:55:57.991 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:55:57.991 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:55:58.342 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:33 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:55:58.501 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:55:58.517 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 264 k 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:55:58.519 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:55:58.519 INFO:teuthology.orchestra.run.smithi038.stdout:Install 2 Packages 2024-03-04T15:55:58.519 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:58.519 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 332 k 2024-03-04T15:55:58.520 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 1.5 M 2024-03-04T15:55:58.520 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:55:58.665 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-04T15:55:58.670 INFO:teuthology.orchestra.run.smithi038.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 454 kB/s | 69 kB 00:00 2024-03-04T15:55:58.727 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-04T15:55:58.779 INFO:teuthology.orchestra.run.smithi038.stdout:(2/2): ceph-volume-19.0.0-1826.g9ca0b5de.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-04T15:55:58.780 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:55:58.780 INFO:teuthology.orchestra.run.smithi038.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-04T15:55:58.781 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:55:58.782 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-04T15:55:58.790 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:55:58.790 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:55:58.819 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-04T15:55:58.836 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:55:58.836 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:55:58.860 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-04T15:55:58.954 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-04T15:55:58.993 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:55:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-04T15:55:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-04T15:55:59.175 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T15:55:59.207 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2/2 2024-03-04T15:55:59.222 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2/2 2024-03-04T15:55:59.223 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:55:59.223 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T15:55:59.223 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:55:59.824 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-04T15:55:59.868 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-04T15:55:59.918 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-04T15:55:59.948 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/2 2024-03-04T15:55:59.966 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-04T15:55:59.988 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 13/13 2024-03-04T15:56:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T15:56:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:00.293 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:56:00.293 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T15:56:00.294 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:00.294 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:00.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados-devel 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 13/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-04T15:56:00.749 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-04T15:56:00.750 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-rook-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-04T15:56:01.128 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:01.129 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:01.245 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:01.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-mgr-cephadm 2024-03-04T15:56:01.421 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 127 k 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:01.422 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:56:01.423 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:01.423 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 127 k 2024-03-04T15:56:01.423 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 456 k 2024-03-04T15:56:01.423 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:56:01.735 INFO:teuthology.orchestra.run.smithi038.stdout:librados-devel-19.0.0-1826.g9ca0b5de.el9.x86_64 405 kB/s | 127 kB 00:00 2024-03-04T15:56:01.736 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:01.736 INFO:teuthology.orchestra.run.smithi038.stdout:Total 403 kB/s | 127 kB 00:00 2024-03-04T15:56:01.736 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:56:01.742 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:56:01.742 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:56:01.800 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:56:01.800 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:56:01.936 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:22 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:56:02.054 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:02.104 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 138 k 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:56:02.105 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout:======================================================================================== 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout:Install 6 Packages 2024-03-04T15:56:02.106 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:02.107 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 6.8 M 2024-03-04T15:56:02.107 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 33 M 2024-03-04T15:56:02.107 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:02.469 INFO:teuthology.orchestra.run.smithi176.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.1 MB/s | 227 kB 00:00 2024-03-04T15:56:02.489 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:02.653 INFO:teuthology.orchestra.run.smithi176.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-04T15:56:02.720 INFO:teuthology.orchestra.run.smithi176.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1826.g9ca0b5de.e 303 kB/s | 138 kB 00:00 2024-03-04T15:56:02.832 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:02.833 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:02.833 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:02.833 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:02.833 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:02.833 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:03.003 INFO:teuthology.orchestra.run.smithi176.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-04T15:56:03.045 INFO:teuthology.orchestra.run.smithi176.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 953 kB/s | 548 kB 00:00 2024-03-04T15:56:03.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs2 2024-03-04T15:56:03.095 INFO:teuthology.orchestra.run.smithi176.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 145 kB/s | 51 kB 00:00 2024-03-04T15:56:03.099 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:03.099 INFO:teuthology.orchestra.run.smithi176.stdout:Total 6.8 MB/s | 6.8 MB 00:00 2024-03-04T15:56:03.100 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:03.108 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:03.109 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:03.199 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:03.200 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:03.511 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:03.655 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:38 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:03.678 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-04T15:56:03.739 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-04T15:56:03.775 INFO:teuthology.orchestra.run.smithi038.stdout:Package libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:03.831 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:03.832 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:03.833 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:03.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install libcephfs-devel 2024-03-04T15:56:04.258 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-04T15:56:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-04T15:56:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-04T15:56:04.426 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 6/6 2024-03-04T15:56:04.462 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:39 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:04.637 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:04.637 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:04.637 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-03-04T15:56:04.637 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 31 k 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:04.638 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 31 k 2024-03-04T15:56:04.639 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 140 k 2024-03-04T15:56:04.639 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:56:04.888 INFO:teuthology.orchestra.run.smithi038.stdout:libcephfs-devel-19.0.0-1826.g9ca0b5de.el9.x86_6 125 kB/s | 31 kB 00:00 2024-03-04T15:56:04.889 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:04.889 INFO:teuthology.orchestra.run.smithi038.stdout:Total 124 kB/s | 31 kB 00:00 2024-03-04T15:56:04.889 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:56:04.893 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:56:04.893 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:56:04.931 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:56:04.932 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:56:05.051 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:56:05.149 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 6/6 2024-03-04T15:56:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/6 2024-03-04T15:56:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-04T15:56:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-04T15:56:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-04T15:56:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-04T15:56:05.541 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-04T15:56:05.541 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-cephadm-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:05.542 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:05.553 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:05.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-fuse 2024-03-04T15:56:05.841 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:05.842 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:05.842 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:05.842 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:05.842 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:05.842 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:06.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librados2 2024-03-04T15:56:06.361 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:27 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:06.527 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 888 k 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:56:06.528 INFO:teuthology.orchestra.run.smithi176.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 966 k 2024-03-04T15:56:06.529 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 2.8 M 2024-03-04T15:56:06.530 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:06.611 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:06.680 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 523 kB/s | 79 kB 00:00 2024-03-04T15:56:06.731 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:06.786 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:06.787 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:06.788 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:06.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install librbd1 2024-03-04T15:56:06.872 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-fuse-19.0.0-1826.g9ca0b5de.el9.x86_ 2.5 MB/s | 888 kB 00:00 2024-03-04T15:56:06.873 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:06.873 INFO:teuthology.orchestra.run.smithi176.stdout:Total 2.7 MB/s | 966 kB 00:00 2024-03-04T15:56:06.874 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:06.894 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:06.894 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:06.970 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:06.970 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:07.156 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:07.290 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-04T15:56:07.363 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/2 2024-03-04T15:56:07.416 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:07.535 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:07.590 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:07.591 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:07.591 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:07.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rados 2024-03-04T15:56:08.029 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2/2 2024-03-04T15:56:08.029 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/2 2024-03-04T15:56:08.218 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:08.336 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:08.377 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:08.391 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:08.392 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:08.392 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:08.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rgw 2024-03-04T15:56:08.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install ceph-volume 2024-03-04T15:56:09.016 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:09.135 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:09.190 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:09.191 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:09.191 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:09.242 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:30 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:09.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-cephfs 2024-03-04T15:56:09.411 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume noarch 2:19.0.0-1826.g9ca0b5de.el9 ceph-noarch 264 k 2024-03-04T15:56:09.412 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:Install 2 Packages 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 332 k 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 1.5 M 2024-03-04T15:56:09.413 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:09.568 INFO:teuthology.orchestra.run.smithi176.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 445 kB/s | 69 kB 00:00 2024-03-04T15:56:09.776 INFO:teuthology.orchestra.run.smithi176.stdout:(2/2): ceph-volume-19.0.0-1826.g9ca0b5de.el9.no 726 kB/s | 264 kB 00:00 2024-03-04T15:56:09.778 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:09.778 INFO:teuthology.orchestra.run.smithi176.stdout:Total 909 kB/s | 332 kB 00:00 2024-03-04T15:56:09.778 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:09.786 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:09.787 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:09.825 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:44 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:09.829 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:09.830 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:09.943 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:09.985 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:09.998 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:09.999 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:09.999 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:10.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install python3-rbd 2024-03-04T15:56:10.176 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-04T15:56:10.212 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2/2 2024-03-04T15:56:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2/2 2024-03-04T15:56:10.228 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:56:10.228 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-04T15:56:10.228 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:10.627 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:45 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:10.746 INFO:teuthology.orchestra.run.smithi038.stdout:Package python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:10.801 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:10.802 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2024-03-04T15:56:10.802 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:10.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-fuse 2024-03-04T15:56:10.974 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 1/2 2024-03-04T15:56:11.295 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-04T15:56:11.295 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:11.295 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:11.295 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-volume-2:19.0.0-1826.g9ca0b5de.el9.noarch 2024-03-04T15:56:11.295 INFO:teuthology.orchestra.run.smithi176.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-04T15:56:11.296 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:11.296 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:11.431 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:46 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:11.606 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 85 k 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:56:11.607 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:11.608 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 85 k 2024-03-04T15:56:11.608 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 231 k 2024-03-04T15:56:11.608 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:56:11.616 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados-devel 2024-03-04T15:56:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-fuse-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 277 kB/s | 85 kB 00:00 2024-03-04T15:56:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:Total 276 kB/s | 85 kB 00:00 2024-03-04T15:56:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:56:11.925 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:56:11.925 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:56:11.982 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:56:11.983 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:56:12.128 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:56:12.190 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:33 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:12.235 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:12.357 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 127 k 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:12.358 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:56:12.359 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:12.359 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 127 k 2024-03-04T15:56:12.359 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 456 k 2024-03-04T15:56:12.359 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:12.590 INFO:teuthology.orchestra.run.smithi176.stdout:librados-devel-19.0.0-1826.g9ca0b5de.el9.x86_64 548 kB/s | 127 kB 00:00 2024-03-04T15:56:12.591 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:12.591 INFO:teuthology.orchestra.run.smithi176.stdout:Total 544 kB/s | 127 kB 00:00 2024-03-04T15:56:12.591 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:12.596 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:12.596 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:12.639 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:12.651 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:12.651 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:12.793 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:12.904 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:12.966 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:12.966 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:12.967 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:12.967 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:13.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-mirror 2024-03-04T15:56:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout: librados-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:13.700 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:13.774 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:48 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:13.941 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs2 2024-03-04T15:56:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repo Size 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.1 M 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:56:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:56:13.950 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 3.1 M 2024-03-04T15:56:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 13 M 2024-03-04T15:56:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:56:14.487 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-mirror-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-04T15:56:14.488 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:14.488 INFO:teuthology.orchestra.run.smithi038.stdout:Total 5.7 MB/s | 3.1 MB 00:00 2024-03-04T15:56:14.488 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:56:14.498 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:56:14.498 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:56:14.514 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:35 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:14.574 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:56:14.574 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:56:14.633 INFO:teuthology.orchestra.run.smithi176.stdout:Package libcephfs2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:14.687 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:14.688 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:14.688 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:14.774 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install libcephfs-devel 2024-03-04T15:56:15.015 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:56:15.079 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:15.104 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:15.104 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:56:15.104 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T15:56:15.104 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:56:15.105 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:56:15.105 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:15.334 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:36 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:15.503 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:15.504 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:15.504 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-04T15:56:15.504 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:15.504 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:15.504 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 31 k 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 31 k 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 140 k 2024-03-04T15:56:15.505 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:15.662 INFO:teuthology.orchestra.run.smithi176.stdout:libcephfs-devel-19.0.0-1826.g9ca0b5de.el9.x86_6 199 kB/s | 31 kB 00:00 2024-03-04T15:56:15.662 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:15.663 INFO:teuthology.orchestra.run.smithi176.stdout:Total 197 kB/s | 31 kB 00:00 2024-03-04T15:56:15.663 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:15.666 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:15.666 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:15.698 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:15.699 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:15.828 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:15.927 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:16.248 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:16.308 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:16.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install rbd-nbd 2024-03-04T15:56:16.593 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:16.593 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:16.593 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:16.593 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-devel-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:16.594 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:16.594 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:16.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librados2 2024-03-04T15:56:17.085 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:52 ago on Mon 04 Mar 2024 03:54:25 PM UTC. 2024-03-04T15:56:17.260 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-04T15:56:17.261 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:17.261 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 170 k 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Install 1 Package 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 170 k 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 490 k 2024-03-04T15:56:17.262 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-04T15:56:17.341 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:38 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:17.456 INFO:teuthology.orchestra.run.smithi176.stdout:Package librados2-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:17.481 INFO:teuthology.orchestra.run.smithi038.stdout:rbd-nbd-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 777 kB/s | 170 kB 00:00 2024-03-04T15:56:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:Total 773 kB/s | 170 kB 00:00 2024-03-04T15:56:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-04T15:56:17.491 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-04T15:56:17.491 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-04T15:56:17.507 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:17.508 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:17.509 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:17.550 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-04T15:56:17.550 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-04T15:56:17.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install librbd1 2024-03-04T15:56:17.710 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-04T15:56:17.825 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:18.120 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:18.235 INFO:teuthology.orchestra.run.smithi176.stdout:Package librbd1-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:18.278 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:18.287 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:18.288 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:18.288 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:18.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rados 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:56:18.574 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-04T15:56:18.836 DEBUG:teuthology.parallel:result is None 2024-03-04T15:56:18.907 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:39 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:19.023 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rados-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:19.076 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:19.077 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:19.077 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:19.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rgw 2024-03-04T15:56:19.701 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:40 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:19.814 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rgw-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:19.866 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:19.867 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:19.867 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:19.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-cephfs 2024-03-04T15:56:20.478 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:41 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:20.592 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-cephfs-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:20.643 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:20.644 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:20.644 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:20.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install python3-rbd 2024-03-04T15:56:21.255 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:42 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:21.370 INFO:teuthology.orchestra.run.smithi176.stdout:Package python3-rbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 is already installed. 2024-03-04T15:56:21.421 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:21.422 INFO:teuthology.orchestra.run.smithi176.stdout:Nothing to do. 2024-03-04T15:56:21.422 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:21.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-fuse 2024-03-04T15:56:22.025 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:43 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:22.192 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 85 k 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:56:22.193 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:22.194 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 85 k 2024-03-04T15:56:22.194 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 231 k 2024-03-04T15:56:22.194 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:22.415 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-fuse-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 387 kB/s | 85 kB 00:00 2024-03-04T15:56:22.415 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:22.415 INFO:teuthology.orchestra.run.smithi176.stdout:Total 385 kB/s | 85 kB 00:00 2024-03-04T15:56:22.416 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:22.422 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:22.422 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:22.478 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:22.479 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:22.619 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:22.722 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:23.118 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-fuse-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:23.468 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:23.701 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-mirror 2024-03-04T15:56:24.296 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:45 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repo Size 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 3.1 M 2024-03-04T15:56:24.467 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 3.1 M 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 13 M 2024-03-04T15:56:24.468 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:25.084 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-mirror-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-04T15:56:25.085 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:25.085 INFO:teuthology.orchestra.run.smithi176.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-04T15:56:25.085 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:25.094 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:25.095 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:25.168 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:25.168 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:25.602 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:25.685 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:25.686 INFO:teuthology.orchestra.run.smithi176.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-04T15:56:25.686 INFO:teuthology.orchestra.run.smithi176.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-04T15:56:25.686 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:56:25.686 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-04T15:56:25.686 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-mirror-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:26.796 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:27.037 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum -y install rbd-nbd 2024-03-04T15:56:27.617 INFO:teuthology.orchestra.run.smithi176.stdout:Last metadata expiration check: 0:01:48 ago on Mon 04 Mar 2024 03:54:39 PM UTC. 2024-03-04T15:56:27.789 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-03-04T15:56:27.789 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: Package Architecture Version Repository Size 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd x86_64 2:19.0.0-1826.g9ca0b5de.el9 ceph 170 k 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Install 1 Package 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 170 k 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 490 k 2024-03-04T15:56:27.790 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-03-04T15:56:28.012 INFO:teuthology.orchestra.run.smithi176.stdout:rbd-nbd-19.0.0-1826.g9ca0b5de.el9.x86_64.rpm 769 kB/s | 170 kB 00:00 2024-03-04T15:56:28.013 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:56:28.013 INFO:teuthology.orchestra.run.smithi176.stdout:Total 763 kB/s | 170 kB 00:00 2024-03-04T15:56:28.013 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-03-04T15:56:28.021 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-03-04T15:56:28.022 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-03-04T15:56:28.077 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-03-04T15:56:28.077 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-03-04T15:56:28.233 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-03-04T15:56:28.344 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:28.819 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:29.129 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 1/1 2024-03-04T15:56:29.129 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:29.130 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-03-04T15:56:29.130 INFO:teuthology.orchestra.run.smithi176.stdout: rbd-nbd-2:19.0.0-1826.g9ca0b5de.el9.x86_64 2024-03-04T15:56:29.130 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:56:29.130 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-03-04T15:56:29.375 DEBUG:teuthology.parallel:result is None 2024-03-04T15:56:29.376 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:56:29.657 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T15:56:29.681 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-1826.g9ca0b5de.el9 2024-03-04T15:56:29.684 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1826.g9ca0b5de.el9 2024-03-04T15:56:29.684 INFO:teuthology.task.install:The correct ceph version 19.0.0-1826.g9ca0b5de is installed. 2024-03-04T15:56:29.686 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:56:29.869 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-04T15:56:29.891 INFO:teuthology.orchestra.run.smithi176.stdout:19.0.0-1826.g9ca0b5de.el9 2024-03-04T15:56:29.891 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1826.g9ca0b5de.el9 2024-03-04T15:56:29.892 INFO:teuthology.task.install:The correct ceph version 19.0.0-1826.g9ca0b5de is installed. 2024-03-04T15:56:29.893 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-04T15:56:29.894 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:56:29.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T15:56:29.926 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:56:29.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-04T15:56:29.979 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-04T15:56:29.980 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:56:29.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T15:56:30.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T15:56:30.080 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:56:30.080 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-03-04T15:56:30.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-04T15:56:30.175 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-04T15:56:30.176 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:56:30.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T15:56:30.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T15:56:30.274 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:56:30.275 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-04T15:56:30.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-04T15:56:30.369 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-04T15:56:30.369 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:56:30.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T15:56:30.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T15:56:30.468 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:56:30.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-03-04T15:56:30.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-04T15:56:30.561 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-04T15:56:30.667 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'cephadm_mode': 'root'} 2024-03-04T15:56:30.667 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:56:30.668 INFO:tasks.cephadm:Cluster fsid is c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:56:30.668 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-04T15:56:30.668 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.38:6789]', 'mon.c': '[v1:172.21.15.38:6790]', 'mon.b': '[v1:172.21.15.176:6789]'} 2024-03-04T15:56:30.668 INFO:tasks.cephadm:First mon is mon.a on smithi038 2024-03-04T15:56:30.668 INFO:tasks.cephadm:First mgr is y 2024-03-04T15:56:30.668 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-04T15:56:30.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-03-04T15:56:30.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-03-04T15:56:30.728 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-04T15:56:30.728 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:56:30.949 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77294/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1826-g9ca0b5de', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1826.g9ca0b5de'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-04 12:37:35.725599', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-asm0deuz-testing-2024-03-04-1211', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-04T15:56:31.177 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-asm0deuz-testing-2024-03-04-1211, sha1 9ca0b5de35ed230e84387abdd8e619ed8e4f3282 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:56:31.179 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm 2024-03-04T15:56:31.179 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:31.588 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:56 /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:31.589 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-asm0deuz-testing-2024-03-04-1211/9ca0b5de35ed230e84387abdd8e619ed8e4f3282/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:32.016 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-r--r--. 1 ubuntu ubuntu 767364 Mar 4 15:56 /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:32.016 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:32.035 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-04T15:56:32.066 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 on all hosts... 2024-03-04T15:56:32.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:56:32.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 pull 2024-03-04T15:56:32.941 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:56:33.022 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:57:02.535 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T15:57:02.535 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:57:02.535 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:57:02.535 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-03-04T15:57:02.535 INFO:teuthology.orchestra.run.smithi176.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:57:02.536 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T15:57:02.536 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev)", 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692", 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977" 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T15:57:06.166 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T15:57:06.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-03-04T15:57:06.244 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-03-04T15:57:06.281 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-03-04T15:57:06.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-03-04T15:57:06.351 INFO:tasks.cephadm:Writing seed config... 2024-03-04T15:57:06.352 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-04T15:57:06.352 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-04T15:57:06.352 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-04T15:57:06.352 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-04T15:57:06.352 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-04T15:57:06.353 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-04T15:57:06.354 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-04T15:57:06.354 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-04T15:57:06.354 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-04T15:57:06.354 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-04T15:57:06.354 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-04T15:57:06.354 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:57:06.355 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-04T15:57:06.369 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c453baf0-da3f-11ee-95c6-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-04T15:57:06.369 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service 2024-03-04T15:57:06.412 DEBUG:teuthology.orchestra.run.smithi038:mgr.y> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y.service 2024-03-04T15:57:06.455 INFO:tasks.cephadm:Bootstrapping... 2024-03-04T15:57:06.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 -v bootstrap --fsid c453baf0-da3f-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.38:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:06.650 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-04T15:57:06.650 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282', '-v', 'bootstrap', '--fsid', 'c453baf0-da3f-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.38:6789]', '--skip-admin-label'] 2024-03-04T15:57:06.675 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:57:06.675 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-04T15:57:06.676 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-03-04T15:57:06.698 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:57:06.698 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-03-04T15:57:06.698 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-03-04T15:57:06.705 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:57:06.705 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:57:06.711 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:57:06.711 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:57:06.718 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-04T15:57:06.723 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-04T15:57:06.724 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:57:06.724 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-03-04T15:57:06.746 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-04T15:57:06.746 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-04T15:57:06.746 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-03-04T15:57:06.746 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-03-04T15:57:06.752 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-04T15:57:06.752 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-04T15:57:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-04T15:57:06.758 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:57:06.764 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 139971270307264 on /run/cephadm/c453baf0-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 139971270307264 acquired on /run/cephadm/c453baf0-da3f-11ee-95c6-87774f69a715.lock 2024-03-04T15:57:06.771 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-03-04T15:57:06.772 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:6789], mon addrv is [v1:172.21.15.38:6789] 2024-03-04T15:57:06.774 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-03-04T15:57:06.774 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-03-04T15:57:06.777 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-04T15:57:06.777 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-04T15:57:06.777 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-04T15:57:06.779 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-03-04T15:57:06.780 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-04T15:57:06.781 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-04T15:57:06.781 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282... 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:64fd3420507ff7dcf06d181c53a3dd949a37f619fe84875331c583b3b5f6f668 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:7feca07754707458c3945cf0062cf4dabc512f6d90fe1a9a1370b362b6011124 2024-03-04T15:57:07.250 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 2024-03-04T15:57:07.251 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-04T15:57:08.214 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:57:08.215 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T15:57:08.215 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-03-04T15:57:09.145 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-03-04T15:57:09.145 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-03-04T15:57:10.201 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQBV7+VlVGklHBAA7v0xOHJtR6S9uMAR59IBng== 2024-03-04T15:57:11.165 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQBW7+VlqPdlIhAAnys8iqgc4dokTPPPovnl6w== 2024-03-04T15:57:12.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQBX7+Vl8TLLGxAAHnci1+smyf/TBvEQh0oOVw== 2024-03-04T15:57:12.014 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-03-04T15:57:12.910 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:57:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-04T15:57:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:12.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:57:12.912 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-03-04T15:57:13.766 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2024-03-04T15:57:13.970 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-04T15:57:14.147 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-04T15:57:14.319 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:57:14.320 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target. 2024-03-04T15:57:14.574 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a 2024-03-04T15:57:14.575 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-04T15:57:14.738 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target.wants/ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:57:15.298 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:57:15.298 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-04T15:57:15.305 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:57:15.305 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:57:15.305 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-04T15:57:15.305 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:57:15.305 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-03-04T15:57:15.306 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.563314s) 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-03-04T15:57:16.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-03-04T15:57:16.306 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-03-04T15:57:17.361 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.38:6789] 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:17.362 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:57:17.363 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:57:17.363 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:57:17.363 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-03-04T15:57:18.375 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-03-04T15:57:18.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:18 smithi038 systemd[1]: Stopping Ceph mon.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:57:18.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29399]: 2024-03-04T15:57:18.640+0000 7fdf514c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T15:57:18.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29399]: 2024-03-04T15:57:18.640+0000 7fdf514c7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-04T15:57:18.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:18 smithi038 podman[29647]: 2024-03-04 15:57:18.835086452 +0000 UTC m=+0.297880033 container died b91f3744eba89eea3f6411476e9bb7b6a60d48e09b869e71a869b959007b331b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T15:57:19.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29647]: 2024-03-04 15:57:19.173944246 +0000 UTC m=+0.636737826 container cleanup b91f3744eba89eea3f6411476e9bb7b6a60d48e09b869e71a869b959007b331b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T15:57:19.435 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 bash[29647]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a 2024-03-04T15:57:19.435 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29661]: 2024-03-04 15:57:19.333363958 +0000 UTC m=+0.499452445 container remove b91f3744eba89eea3f6411476e9bb7b6a60d48e09b869e71a869b959007b331b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:57:19.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-04T15:57:19.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 systemd[1]: Stopped Ceph mon.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:57:19.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 systemd[1]: Starting Ceph mon.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29760]: 2024-03-04 15:57:19.767610641 +0000 UTC m=+0.107007229 container create 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29760]: 2024-03-04 15:57:19.678220249 +0000 UTC m=+0.017616850 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29760]: 2024-03-04 15:57:19.970081917 +0000 UTC m=+0.309478509 container init 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 podman[29760]: 2024-03-04 15:57:19.972754976 +0000 UTC m=+0.312151567 container start 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: pidfile_write: ignore empty --pid-file 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: load: jerasure load: lrc 2024-03-04T15:57:20.030 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: Git sha 0 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: DB SUMMARY 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: DB Session ID: GZOAUZVGO0HDKUGJ9ZTD 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89546 ; 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:19 smithi038 ceph-mon[29779]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:57:20.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.env: 0x55e2393a5b20 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.info_log: 0x55e23b767920 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.statistics: (nil) 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.use_fsync: 0 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:57:20.032 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.db_log_dir: 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.wal_dir: 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:57:20.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.write_buffer_manager: 0x55e23b73b400 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:57:20.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:57:20.035 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:57:20.035 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:57:20.035 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:57:20.035 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.unordered_write: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.row_cache: None 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.wal_filter: None 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.wal_compression: 0 2024-03-04T15:57:20.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:57:20.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_open_files: -1 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:57:20.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Compression algorithms supported: 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kZSTD supported: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:57:20.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.merge_operator: 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_filter: None 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e23b767a20) 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:57:20.040 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x55e23b76d090 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2024-03-04T15:57:20.041 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:57:20.042 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression: NoCompression 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:57:20.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.num_levels: 7 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:57:20.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:57:20.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:57:20.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:57:20.047 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:57:20.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.ttl: 2592000 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:57:20.049 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f1824859-df21-4d73-a836-c4c9c42b5a25 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567840000856, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567840001817, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85483, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 83591, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11691, "raw_average_key_size": 48, "raw_value_size": 77146, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709567840, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f1824859-df21-4d73-a836-c4c9c42b5a25", "db_session_id": "GZOAUZVGO0HDKUGJ9ZTD", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567840001912, "job": 1, "event": "recovery_finished"} 2024-03-04T15:57:20.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e23b860000 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: DB pointer 0x55e23b84a000 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:57:20.051 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 85.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 85.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.052 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 14.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 14.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x55e23b76d090#2 capacity: 512.00 MB usage: 1.25 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.053 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: starting mon.a rank 0 at public addrs v1:172.21.15.38:6789/0 at bind addrs v1:172.21.15.38:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???) e1 preinit fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).mds e1 new map 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).mds e1 print_map 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:20.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:57:20.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-04T15:57:20.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 bash[29760]: 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 2024-03-04T15:57:20.055 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 systemd[1]: Started Ceph mon.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:57:20.055 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-04T15:57:20.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-04T15:57:20.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: monmap e1: 1 mons at {a=v1:172.21.15.38:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:57:20.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: fsmap 2024-03-04T15:57:20.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: osdmap e1: 0 total, 0 up, 0 in 2024-03-04T15:57:20.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:20 smithi038 ceph-mon[29779]: mgrmap e1: no daemons active 2024-03-04T15:57:21.156 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-04T15:57:21.177 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:57:21.177 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-03-04T15:57:21.178 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-04T15:57:21.178 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-04T15:57:21.422 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y 2024-03-04T15:57:21.423 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y.service: Unit ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y.service not loaded. 2024-03-04T15:57:21.565 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715.target.wants/ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c453baf0-da3f-11ee-95c6-87774f69a715@.service. 2024-03-04T15:57:21.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4269745003' entity='client.admin' 2024-03-04T15:57:22.049 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:21 smithi038 podman[29994]: 2024-03-04 15:57:21.966853322 +0000 UTC m=+0.285024793 container start 5d1ee81947b483bf2e552e81877c03effbbfc053a498cd59d51d08d9527a4460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2) 2024-03-04T15:57:22.049 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:22 smithi038 bash[29994]: 5d1ee81947b483bf2e552e81877c03effbbfc053a498cd59d51d08d9527a4460 2024-03-04T15:57:22.062 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:57:22.062 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-04T15:57:22.069 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:57:22.069 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:57:22.069 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-04T15:57:22.069 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-04T15:57:22.077 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:57:22.077 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-03-04T15:57:22.085 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-03-04T15:57:22.301 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:22 smithi038 systemd[1]: Started Ceph mgr.y for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:57:22.592 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:22 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:22.531+0000 7fb718dd2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:57:22.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/401360994' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:57:22.897 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:22 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:22.631+0000 7fb718dd2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:57:23.125 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:23.127 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:23.127 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:57:23.127 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:23.128 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:23.129 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:57:23.130 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:57:23.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:57:23.134 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:57:23.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:57:23.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:57:15.256033+0000", 2024-03-04T15:57:23.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:57:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:23.138 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-03-04T15:57:23.630 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.294+0000 7fb718dd2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:57:23.630 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.389+0000 7fb718dd2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:57:23.630 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.532+0000 7fb718dd2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:57:23.981 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.628+0000 7fb718dd2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:57:23.981 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.712+0000 7fb718dd2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:57:23.981 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:23.979+0000 7fb718dd2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:57:24.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:24 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:24.091+0000 7fb718dd2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:24.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:24 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:24.179+0000 7fb718dd2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:57:24.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:24 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:24.275+0000 7fb718dd2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:57:24.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:24 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:24.455+0000 7fb718dd2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:57:25.558 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:25.460+0000 7fb718dd2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:57:25.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1520790700' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:57:25.941 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:25.598+0000 7fb718dd2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:57:25.941 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:25.938+0000 7fb718dd2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:57:26.232 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:26 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:26.031+0000 7fb718dd2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:57:26.232 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:26 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:26.230+0000 7fb718dd2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:26.241 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:26.242 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:57:26.243 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:57:26.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:57:26.244 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:57:26.247 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:57:26.248 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:57:15.256033+0000", 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:26.249 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2024-03-04T15:57:26.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:26 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:26.419+0000 7fb718dd2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:57:26.973 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:26 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:26.710+0000 7fb718dd2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:26.973 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:26 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:26.971+0000 7fb718dd2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:27.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:27 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:27.211+0000 7fb718dd2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:57:28.362 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:28 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:28.031+0000 7fb718dd2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:57:28.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:28 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/822397058' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:57:28.953 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:28 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:28.621+0000 7fb718dd2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:57:28.953 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:28 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:28.726+0000 7fb718dd2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:57:28.953 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:28 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:28.950+0000 7fb718dd2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:57:30.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: Activating manager daemon y 2024-03-04T15:57:30.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: mgrmap e2: y(active, starting, since 0.00483935s) 2024-03-04T15:57:30.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: Manager daemon y is now available 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' 2024-03-04T15:57:30.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:29 smithi038 ceph-mon[29779]: from='mgr.14100 v1:172.21.15.38:0/603439983' entity='mgr.y' 2024-03-04T15:57:31.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:30 smithi038 ceph-mon[29779]: mgrmap e3: y(active, since 1.00911s) 2024-03-04T15:57:32.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:32.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:32.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:57:32.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:57:32.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:57:32.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:32.432 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:57:15.256033+0000", 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:32.433 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (3/15)... 2024-03-04T15:57:33.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:32 smithi038 ceph-mon[29779]: mgrmap e4: y(active, since 3s) 2024-03-04T15:57:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2335864146' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-04T15:57:39.652 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-04T15:57:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-04T15:57:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-04T15:57:39.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:57:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-04T15:57:15.256033+0000", 2024-03-04T15:57:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-04T15:57:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-04T15:57:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-04T15:57:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-03-04T15:57:40.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1078238078' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-04T15:57:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-04T15:57:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:40.861 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-04T15:57:40.862 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-03-04T15:57:41.938 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/251941705' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-04T15:57:41.938 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setuser ceph since I am not root 2024-03-04T15:57:41.939 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setgroup ceph since I am not root 2024-03-04T15:57:42.447 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:42 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:42.191+0000 7f01a44c6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:57:42.447 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:42 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:42.289+0000 7f01a44c6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:57:42.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:42 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/251941705' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-04T15:57:42.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:42 smithi038 ceph-mon[29779]: mgrmap e5: y(active, since 12s) 2024-03-04T15:57:43.198 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:42 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:42.958+0000 7f01a44c6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:57:43.198 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.053+0000 7f01a44c6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:57:43.198 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.196+0000 7f01a44c6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:57:43.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.294+0000 7f01a44c6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:57:43.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.379+0000 7f01a44c6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:57:43.922 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3902411813' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:57:43.922 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.648+0000 7f01a44c6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:57:43.922 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.760+0000 7f01a44c6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:43.922 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.848+0000 7f01a44c6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:57:44.227 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:43.945+0000 7f01a44c6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:57:44.228 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:44 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:44.126+0000 7f01a44c6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-04T15:57:44.434 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-03-04T15:57:45.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:45 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:45.150+0000 7f01a44c6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:57:45.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:45 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:45.286+0000 7f01a44c6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:57:45.917 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:45 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:45.624+0000 7f01a44c6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:57:45.917 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:45 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:45.716+0000 7f01a44c6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:57:45.918 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:45 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:45.916+0000 7f01a44c6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:46.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:46 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:46.104+0000 7f01a44c6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:57:46.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:46 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:46.395+0000 7f01a44c6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:57:46.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:46 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:46.658+0000 7f01a44c6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:57:46.994 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:46 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:46.900+0000 7f01a44c6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:57:47.994 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:47 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:47.722+0000 7f01a44c6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:57:48.628 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:48.305+0000 7f01a44c6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:57:48.628 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:48.406+0000 7f01a44c6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:57:48.628 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:57:48.626+0000 7f01a44c6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: Active manager daemon y restarted 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: Activating manager daemon y 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: osdmap e2: 0 total, 0 up, 0 in 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: mgrmap e6: y(active, starting, since 0.00801316s) 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:57:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:57:48.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: Manager daemon y is now available 2024-03-04T15:57:48.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:48.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:48 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: Found migration_current of "None". Setting to last migration. 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='client.14124 v1:172.21.15.38:0/1665290878' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: mgrmap e7: y(active, since 1.01424s) 2024-03-04T15:57:49.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:49 smithi038 ceph-mon[29779]: from='client.14124 v1:172.21.15.38:0/1665290878' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:57:50.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:57:50.256 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-04T15:57:50.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:57:50.257 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:57:50.257 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-03-04T15:57:50.257 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-03-04T15:57:51.176 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:51 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:51.176 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:51 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:51.177 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:51 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:51.177 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:51 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:51.177 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:51 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:57:52.142 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: [04/Mar/2024:15:57:50] ENGINE Bus STARTING 2024-03-04T15:57:52.142 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: [04/Mar/2024:15:57:50] ENGINE Serving on https://172.21.15.38:7150 2024-03-04T15:57:52.142 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: from='client.14132 v1:172.21.15.38:0/1641567372' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:52.142 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: [04/Mar/2024:15:57:50] ENGINE Serving on http://172.21.15.38:8765 2024-03-04T15:57:52.142 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: [04/Mar/2024:15:57:50] ENGINE Bus STARTED 2024-03-04T15:57:52.143 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:52 smithi038 ceph-mon[29779]: mgrmap e8: y(active, since 2s) 2024-03-04T15:57:52.745 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-04T15:57:52.745 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-03-04T15:57:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-mon[29779]: from='client.14134 v1:172.21.15.38:0/2737680267' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:54.173 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:54 smithi038 ceph-mon[29779]: from='client.14136 v1:172.21.15.38:0/674608443' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:54.173 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:54 smithi038 ceph-mon[29779]: Generating ssh key... 2024-03-04T15:57:54.173 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:54 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:54.173 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:54 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: Generating public/private rsa key pair. 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: Your identification has been saved in /tmp/tmppt4s_ygl/key. 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: Your public key has been saved in /tmp/tmppt4s_ygl/key.pub. 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: The key fingerprint is: 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: SHA256:GKIxwFc7Y6F6WYG0SWrUr5vLUCnZBxGNzr+5Q4nn2vk ceph-c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: The key's randomart image is: 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: +---[RSA 3072]----+ 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: |o.o=*+ | 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: |.oo==.+ | 2024-03-04T15:57:54.174 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | o*=o*. | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: |. +**ooo | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | +.*+.o S | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | +o.= | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | . * o | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | oo.=. | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: | +o++E | 2024-03-04T15:57:54.175 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:57:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: +----[SHA256]-----+ 2024-03-04T15:57:55.701 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkN98HgPIOx6HdNYacnXPhFgDJemtDFBKiQHGyPSk57odPBKfOnheFoWRyGfiIdk4njjjzTXeCmtTzK1qj+i5xl4XNA2Wen1qnpCa8HNFBf5gb+GWEbI6P+lUIui4Q9CTm0Yxmg2xpp6pWPaRgSlcKJfcU6hbgDaatowAb27QHJp7UMr5soyPE3fRfw7cE5hM/lkcjyzyQzlbwNeMdz9sonrkS9tOcbyjKexcx5AynW3giDXUaWiwts7TtzVIIMkcrbIy2RMnlBdpffS0YBt9zKw+TD0KW/SFxSvAiXHMYswwzGQ08S7MumRWlLu8kIjRZmvSqYzq3uTO8PzIzd1Pf/t9A3v9wAO4W81cqUWklTkSUX4xqKKDaelYUAloF3p7utr65TZ7HDijUTtvAX+MrhObyXc5Hgkv45ddjS3kpc+ytv0/3apFOW4/4Oam0k6qr93Yu8V6un5/hrQOjr8ht4+nKrON3zia4HL1QxfNVSBiCNSp9iMiYiF5tmfaYlVE= ceph-c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:57:55.702 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-04T15:57:55.702 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-03-04T15:57:55.702 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-03-04T15:57:55.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:55 smithi038 ceph-mon[29779]: from='client.14138 v1:172.21.15.38:0/92855188' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:56.933 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:56 smithi038 ceph-mon[29779]: from='client.14140 v1:172.21.15.38:0/3388946214' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:57:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:57:57 smithi038 ceph-mon[29779]: Deploying cephadm binary to smithi038 2024-03-04T15:58:01.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-03-04T15:58:01.317 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2024-03-04T15:58:01.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:01 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:01.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:01 smithi038 ceph-mon[29779]: Added host smithi038 2024-03-04T15:58:01.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:01 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:02.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-04T15:58:02.921 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2024-03-04T15:58:03.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:03 smithi038 ceph-mon[29779]: from='client.14142 v1:172.21.15.38:0/3263620666' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:03.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:03 smithi038 ceph-mon[29779]: Saving service mon spec with placement count:5 2024-03-04T15:58:03.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:03 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:04.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-04T15:58:04.637 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:04 smithi038 ceph-mon[29779]: from='client.14144 v1:172.21.15.38:0/3036998946' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:04.637 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:04 smithi038 ceph-mon[29779]: Saving service mgr spec with placement count:2 2024-03-04T15:58:04.637 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:04 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:04.637 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:04 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:06.104 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:05 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:06.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2182342180' entity='client.admin' 2024-03-04T15:58:07.104 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-03-04T15:58:07.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4050000510' entity='client.admin' 2024-03-04T15:58:09.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:08 smithi038 ceph-mon[29779]: from='mgr.14120 v1:172.21.15.38:0/765172331' entity='mgr.y' 2024-03-04T15:58:09.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:08 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1612085227' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-04T15:58:09.054 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:08 smithi038 ceph-mon[29779]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:09.055 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:08 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setuser ceph since I am not root 2024-03-04T15:58:09.055 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:08 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setgroup ceph since I am not root 2024-03-04T15:58:09.621 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:09 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:09.367+0000 7f2ca6a7c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:58:09.621 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:09 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:09.467+0000 7f2ca6a7c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:58:09.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1612085227' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-04T15:58:09.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:09 smithi038 ceph-mon[29779]: mgrmap e9: y(active, since 20s) 2024-03-04T15:58:10.375 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.135+0000 7f2ca6a7c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:58:10.375 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.230+0000 7f2ca6a7c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:58:10.375 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.373+0000 7f2ca6a7c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:58:10.651 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.472+0000 7f2ca6a7c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:58:10.651 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.557+0000 7f2ca6a7c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:58:10.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:58:10.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-04T15:58:10.655 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-03-04T15:58:10.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/467842908' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-04T15:58:10.903 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.834+0000 7f2ca6a7c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:58:11.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:10.950+0000 7f2ca6a7c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:58:11.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:11 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:11.040+0000 7f2ca6a7c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:58:11.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:11 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:11.138+0000 7f2ca6a7c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:58:11.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:11 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:11.317+0000 7f2ca6a7c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:58:12.645 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:12 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:12.305+0000 7f2ca6a7c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:58:12.645 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:12 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:12.439+0000 7f2ca6a7c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:58:12.916 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:12 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:12.780+0000 7f2ca6a7c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:58:12.916 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:12 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:12.875+0000 7f2ca6a7c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:58:13.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:13.082+0000 7f2ca6a7c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:58:13.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:13.278+0000 7f2ca6a7c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:58:13.841 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:13.575+0000 7f2ca6a7c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:58:13.841 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:13.839+0000 7f2ca6a7c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:58:14.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:14.083+0000 7f2ca6a7c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:58:15.244 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:14.912+0000 7f2ca6a7c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:58:15.835 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:15.509+0000 7f2ca6a7c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:58:15.835 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:15.610+0000 7f2ca6a7c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:58:15.835 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T15:58:15.833+0000 7f2ca6a7c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: Active manager daemon y restarted 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: Activating manager daemon y 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: osdmap e3: 0 total, 0 up, 0 in 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: mgrmap e10: y(active, starting, since 0.006447s) 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T15:58:16.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T15:58:16.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: Manager daemon y is now available 2024-03-04T15:58:16.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:16.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:17.169 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T15:58:17.169 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T15:58:17.169 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: [04/Mar/2024:15:58:16] ENGINE Bus STARTING 2024-03-04T15:58:17.169 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: [04/Mar/2024:15:58:16] ENGINE Serving on http://172.21.15.38:8765 2024-03-04T15:58:17.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: [04/Mar/2024:15:58:16] ENGINE Serving on https://172.21.15.38:7150 2024-03-04T15:58:17.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: [04/Mar/2024:15:58:16] ENGINE Bus STARTED 2024-03-04T15:58:17.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:16 smithi038 ceph-mon[29779]: mgrmap e11: y(active, since 1.00765s) 2024-03-04T15:58:17.422 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-04T15:58:17.422 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-04T15:58:17.422 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-04T15:58:17.423 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-04T15:58:17.423 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-03-04T15:58:17.423 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-03-04T15:58:18.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:17 smithi038 ceph-mon[29779]: from='client.14156 v1:172.21.15.38:0/2833941585' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-04T15:58:18.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:17 smithi038 ceph-mon[29779]: from='client.14156 v1:172.21.15.38:0/2833941585' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-04T15:58:19.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:18 smithi038 ceph-mon[29779]: mgrmap e12: y(active, since 2s) 2024-03-04T15:58:19.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:19.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:18 smithi038 ceph-mon[29779]: from='client.14164 v1:172.21.15.38:0/2858044925' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:19.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:19.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:19.081 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-04T15:58:19.081 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-03-04T15:58:20.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:20.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:20 smithi038 ceph-mon[29779]: from='client.14166 v1:172.21.15.38:0/544712011' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:20.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:21.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$sf0Hd4gTApfGEoPCUr7c2OEr6uv3gYm8y/KkZD4tA4YvXf0IpWbY2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709567900, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-04T15:58:21.354 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-03-04T15:58:22.594 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-03-04T15:58:22.603 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-04T15:58:22.603 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-04T15:58:22.611 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-04T15:58:22.611 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-04T15:58:22.611 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-04T15:58:22.611 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-04T15:58:22.612 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-03-04T15:58:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038.front.sepia.ceph.com:8443/ 2024-03-04T15:58:22.612 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-03-04T15:58:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: Password: 1zfsdi0sld 2024-03-04T15:58:22.613 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:22.613 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config directory 2024-03-04T15:58:22.613 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-03-04T15:58:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:22 smithi038 ceph-mon[29779]: mgrmap e13: y(active, since 6s) 2024-03-04T15:58:23.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3237149945' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.193 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:25.194 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-03-04T15:58:25.234 INFO:tasks.cephadm:Fetching config... 2024-03-04T15:58:25.235 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:58:25.235 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-04T15:58:25.250 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-04T15:58:25.250 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:58:25.251 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-04T15:58:25.305 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-04T15:58:25.305 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:58:25.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-04T15:58:25.372 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-04T15:58:25.373 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:58:25.373 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-04T15:58:25.427 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-04T15:58:25.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkN98HgPIOx6HdNYacnXPhFgDJemtDFBKiQHGyPSk57odPBKfOnheFoWRyGfiIdk4njjjzTXeCmtTzK1qj+i5xl4XNA2Wen1qnpCa8HNFBf5gb+GWEbI6P+lUIui4Q9CTm0Yxmg2xpp6pWPaRgSlcKJfcU6hbgDaatowAb27QHJp7UMr5soyPE3fRfw7cE5hM/lkcjyzyQzlbwNeMdz9sonrkS9tOcbyjKexcx5AynW3giDXUaWiwts7TtzVIIMkcrbIy2RMnlBdpffS0YBt9zKw+TD0KW/SFxSvAiXHMYswwzGQ08S7MumRWlLu8kIjRZmvSqYzq3uTO8PzIzd1Pf/t9A3v9wAO4W81cqUWklTkSUX4xqKKDaelYUAloF3p7utr65TZ7HDijUTtvAX+MrhObyXc5Hgkv45ddjS3kpc+ytv0/3apFOW4/4Oam0k6qr93Yu8V6un5/hrQOjr8ht4+nKrON3zia4HL1QxfNVSBiCNSp9iMiYiF5tmfaYlVE= ceph-c453baf0-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:58:25.505 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkN98HgPIOx6HdNYacnXPhFgDJemtDFBKiQHGyPSk57odPBKfOnheFoWRyGfiIdk4njjjzTXeCmtTzK1qj+i5xl4XNA2Wen1qnpCa8HNFBf5gb+GWEbI6P+lUIui4Q9CTm0Yxmg2xpp6pWPaRgSlcKJfcU6hbgDaatowAb27QHJp7UMr5soyPE3fRfw7cE5hM/lkcjyzyQzlbwNeMdz9sonrkS9tOcbyjKexcx5AynW3giDXUaWiwts7TtzVIIMkcrbIy2RMnlBdpffS0YBt9zKw+TD0KW/SFxSvAiXHMYswwzGQ08S7MumRWlLu8kIjRZmvSqYzq3uTO8PzIzd1Pf/t9A3v9wAO4W81cqUWklTkSUX4xqKKDaelYUAloF3p7utr65TZ7HDijUTtvAX+MrhObyXc5Hgkv45ddjS3kpc+ytv0/3apFOW4/4Oam0k6qr93Yu8V6un5/hrQOjr8ht4+nKrON3zia4HL1QxfNVSBiCNSp9iMiYiF5tmfaYlVE= ceph-c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:58:25.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkN98HgPIOx6HdNYacnXPhFgDJemtDFBKiQHGyPSk57odPBKfOnheFoWRyGfiIdk4njjjzTXeCmtTzK1qj+i5xl4XNA2Wen1qnpCa8HNFBf5gb+GWEbI6P+lUIui4Q9CTm0Yxmg2xpp6pWPaRgSlcKJfcU6hbgDaatowAb27QHJp7UMr5soyPE3fRfw7cE5hM/lkcjyzyQzlbwNeMdz9sonrkS9tOcbyjKexcx5AynW3giDXUaWiwts7TtzVIIMkcrbIy2RMnlBdpffS0YBt9zKw+TD0KW/SFxSvAiXHMYswwzGQ08S7MumRWlLu8kIjRZmvSqYzq3uTO8PzIzd1Pf/t9A3v9wAO4W81cqUWklTkSUX4xqKKDaelYUAloF3p7utr65TZ7HDijUTtvAX+MrhObyXc5Hgkv45ddjS3kpc+ytv0/3apFOW4/4Oam0k6qr93Yu8V6un5/hrQOjr8ht4+nKrON3zia4HL1QxfNVSBiCNSp9iMiYiF5tmfaYlVE= ceph-c453baf0-da3f-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-04T15:58:25.579 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkN98HgPIOx6HdNYacnXPhFgDJemtDFBKiQHGyPSk57odPBKfOnheFoWRyGfiIdk4njjjzTXeCmtTzK1qj+i5xl4XNA2Wen1qnpCa8HNFBf5gb+GWEbI6P+lUIui4Q9CTm0Yxmg2xpp6pWPaRgSlcKJfcU6hbgDaatowAb27QHJp7UMr5soyPE3fRfw7cE5hM/lkcjyzyQzlbwNeMdz9sonrkS9tOcbyjKexcx5AynW3giDXUaWiwts7TtzVIIMkcrbIy2RMnlBdpffS0YBt9zKw+TD0KW/SFxSvAiXHMYswwzGQ08S7MumRWlLu8kIjRZmvSqYzq3uTO8PzIzd1Pf/t9A3v9wAO4W81cqUWklTkSUX4xqKKDaelYUAloF3p7utr65TZ7HDijUTtvAX+MrhObyXc5Hgkv45ddjS3kpc+ytv0/3apFOW4/4Oam0k6qr93Yu8V6un5/hrQOjr8ht4+nKrON3zia4HL1QxfNVSBiCNSp9iMiYiF5tmfaYlVE= ceph-c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:58:25.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-04T15:58:25.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1981128650' entity='client.admin' 2024-03-04T15:58:25.847 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:28.848 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-04T15:58:28.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-04T15:58:29.086 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:58:29.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:29 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1884458936' entity='client.admin' 2024-03-04T15:58:29.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:29.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:31.483 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-03-04T15:58:31.483 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:58:31.483 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-03-04T15:58:31.501 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:58:31.501 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:58:31.561 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-03-04T15:58:31.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch host add smithi176 2024-03-04T15:58:31.816 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='client.14176 v1:172.21.15.38:0/2704918656' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:31.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T15:58:31.938 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:31.938 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:31.938 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:32.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:32 smithi038 ceph-mon[29779]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:58:32.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:32 smithi038 ceph-mon[29779]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:58:32.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:32 smithi038 ceph-mon[29779]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:58:34.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:33 smithi038 ceph-mon[29779]: from='client.14178 v1:172.21.15.38:0/283216721' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:34 smithi038 ceph-mon[29779]: Deploying cephadm binary to smithi176 2024-03-04T15:58:37.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:36 smithi038 ceph-mon[29779]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:38.083 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-03-04T15:58:38.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-04T15:58:38.928 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: Added host smithi176 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:39.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:40.741 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T15:58:40.741 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-03-04T15:58:41.307 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:41 smithi038 ceph-mon[29779]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:41.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:41 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:41.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:41 smithi038 ceph-mon[29779]: from='client.14180 v1:172.21.15.38:0/2773404006' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T15:58:41.337 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-04T15:58:41.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-04T15:58:41.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:58:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:43 smithi038 ceph-mon[29779]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:44.184 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:44.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:58:44.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:58:44.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:44 smithi038 ceph-mon[29779]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:58:44.732 INFO:tasks.cephadm:Adding mon.a on smithi038 2024-03-04T15:58:44.732 INFO:tasks.cephadm:Adding mon.c on smithi038 2024-03-04T15:58:44.732 INFO:tasks.cephadm:Adding mon.b on smithi176 2024-03-04T15:58:44.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b' 2024-03-04T15:58:44.957 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:46.679 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mon update... 2024-03-04T15:58:47.289 DEBUG:teuthology.orchestra.run.smithi038:mon.c> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.c.service 2024-03-04T15:58:47.293 DEBUG:teuthology.orchestra.run.smithi176:mon.b> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.b.service 2024-03-04T15:58:47.296 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:58:47.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='client.14184 v1:172.21.15.176:0/908284901' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: Saving service mon spec with placement smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b;count:3 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:47.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:47.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:47.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:58:47.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:47.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:47 smithi038 ceph-mon[29779]: Deploying daemon mon.b on smithi176 2024-03-04T15:58:49.073 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:58:49.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:49 smithi038 ceph-mon[29779]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:50.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:50 smithi176 systemd[1]: Starting Ceph mon.b for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T15:58:51.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 podman[33560]: 2024-03-04 15:58:50.90851466 +0000 UTC m=+0.549563291 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:58:51.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 podman[33560]: 2024-03-04 15:58:51.012195683 +0000 UTC m=+0.653244316 container create f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T15:58:51.185 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:58:51.185 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:57:12.310111Z","created":"2024-03-04T15:57:12.310111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-04T15:58:51.188 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-03-04T15:58:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 podman[33560]: 2024-03-04 15:58:51.206124045 +0000 UTC m=+0.847172680 container init f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T15:58:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 podman[33560]: 2024-03-04 15:58:51.20990759 +0000 UTC m=+0.850956219 container start f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: pidfile_write: ignore empty --pid-file 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: load: jerasure load: lrc 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Git sha 0 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: DB SUMMARY 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: DB Session ID: IZAMM2TU9MK93KHBVNNP 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:58:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.env: 0x55fb86fdeb20 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.info_log: 0x55fb87f13920 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:58:51.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.statistics: (nil) 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.use_fsync: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:58:51.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.db_log_dir: 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.wal_dir: 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.write_buffer_manager: 0x55fb87ee7400 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:58:51.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.unordered_write: 0 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.row_cache: None 2024-03-04T15:58:51.614 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.wal_filter: None 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.wal_compression: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:58:51.616 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:58:51.617 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_open_files: -1 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:58:51.618 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Compression algorithms supported: 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kZSTD supported: 0 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:58:51.619 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.merge_operator: 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_filter: None 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fb87f13a20) 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:58:51.620 INFO:journalctl@ceph.mon.b.smithi176.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_type: 0 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: data_block_index_type: 0 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_shortening: 1 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: checksum: 4 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: no_block_cache: 0 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache: 0x55fb87f19090 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_options: 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: capacity : 536870912 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: num_shard_bits : 4 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: strict_capacity_limit : 0 2024-03-04T15:58:51.621 INFO:journalctl@ceph.mon.b.smithi176.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_cache_compressed: (nil) 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: persistent_cache: (nil) 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_size: 4096 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_size_deviation: 10 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_restart_interval: 16 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: index_block_restart_interval: 1 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: metadata_block_size: 4096 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: partition_filters: 0 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: use_delta_encoding: 1 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: filter_policy: bloomfilter 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: whole_key_filtering: 1 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: verify_compression: 0 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:58:51.622 INFO:journalctl@ceph.mon.b.smithi176.stdout: format_version: 5 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: enable_index_compression: 1 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: block_align: 0 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: max_auto_readahead_size: 262144 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: prepopulate_block_cache: 0 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression: NoCompression 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:58:51.623 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.num_levels: 7 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:58:51.624 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:58:51.625 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:58:51.626 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:58:51.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:58:51.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.ttl: 2592000 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:58:51.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fbe9031a-8043-4b28-8bd8-eeda1d180251 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567931251311, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567931251736, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709567931, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fbe9031a-8043-4b28-8bd8-eeda1d180251", "db_session_id": "IZAMM2TU9MK93KHBVNNP", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567931251815, "job": 1, "event": "recovery_finished"} 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fb8800c000 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: DB pointer 0x55fb87ff6000 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:58:51.630 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** DB Stats ** 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:58:51.631 INFO:journalctl@ceph.mon.b.smithi176.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** Compaction Stats [default] ** 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:58:51.632 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: Cumulative compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: Interval compaction: 0.00 GB write, 0.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache BinnedLRUCache@0x55fb87f19090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: using public_addrv v1:172.21.15.176:6789/0 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: starting mon.b rank -1 at public addrs v1:172.21.15.176:6789/0 at bind addrs v1:172.21.15.176:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(???) e0 preinit fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:58:51.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).mds e1 new map 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: e1 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: legacy client fscid: -1 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout: No filesystems configured 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:58:51.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.635 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:51.636 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='client.14184 v1:172.21.15.176:0/908284901' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Saving service mon spec with placement smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b;count:3 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: Deploying daemon mon.b on smithi176 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:51.637 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 ceph-mon[33619]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-04T15:58:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 bash[33560]: f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 2024-03-04T15:58:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:51 smithi176 systemd[1]: Started Ceph mon.b for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:58:52.795 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:58:52.795 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:58:53.040 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:58:56.626 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:58:56.627 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":2,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:58:51.272381Z","created":"2024-03-04T15:57:12.310111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-04T15:58:56.627 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: mon.a calling monitor election 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: mon.b calling monitor election 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-04T15:58:56.628 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: monmap e2: 2 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: fsmap 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: mgrmap e13: y(active, since 40s) 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: overall HEALTH_OK 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:58:56.629 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:56.629 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 2 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: mon.a calling monitor election 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: mon.b calling monitor election 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: monmap e2: 2 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: fsmap 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: mgrmap e13: y(active, since 40s) 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: overall HEALTH_OK 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:58:56.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:58:56.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:58:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:57 smithi176 ceph-mon[33619]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:57 smithi176 ceph-mon[33619]: Deploying daemon mon.c on smithi038 2024-03-04T15:58:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/1442846988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:58:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:57 smithi038 ceph-mon[29779]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:57 smithi038 ceph-mon[29779]: Deploying daemon mon.c on smithi038 2024-03-04T15:58:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/1442846988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:58:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:58:58.199 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-04T15:58:58.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-04T15:58:58.448 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:58:59.555 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[29779]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:59.555 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 podman[38085]: 2024-03-04 15:58:59.554010243 +0000 UTC m=+0.153154752 container create 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T15:58:59.659 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:58:59 smithi176 ceph-mon[33619]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:58:59.817 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 podman[38085]: 2024-03-04 15:58:59.789945687 +0000 UTC m=+0.389090197 container init 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0) 2024-03-04T15:58:59.817 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 podman[38085]: 2024-03-04 15:58:59.793022187 +0000 UTC m=+0.392166700 container start 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, GIT_CLEAN=True) 2024-03-04T15:59:00.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: set uid:gid to 167:167 (ceph:ceph) 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev), process ceph-mon, pid 2 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pidfile_write: ignore empty --pid-file 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: load: jerasure load: lrc 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: RocksDB version: 7.9.2 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Git sha 0 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Compile date 2024-03-04 11:37:16 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: DB SUMMARY 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: DB Session ID: Z4XT510EBGK77A9ZI2M3 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: CURRENT file: CURRENT 2024-03-04T15:59:00.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: IDENTITY file: IDENTITY 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.error_if_exists: 0 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.create_if_missing: 0 2024-03-04T15:59:00.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.paranoid_checks: 1 2024-03-04T15:59:00.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-04T15:59:00.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-04T15:59:00.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-04T15:59:00.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.env: 0x55bda8ddbb20 2024-03-04T15:59:00.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.fs: PosixFileSystem 2024-03-04T15:59:00.248 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.info_log: 0x55bda96c9900 2024-03-04T15:59:00.248 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_file_opening_threads: 16 2024-03-04T15:59:00.248 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.statistics: (nil) 2024-03-04T15:59:00.248 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.use_fsync: 0 2024-03-04T15:59:00.248 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_log_file_size: 0 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.keep_log_file_num: 1000 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.recycle_log_file_num: 0 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_fallocate: 1 2024-03-04T15:59:00.249 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_mmap_reads: 0 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_mmap_writes: 0 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.use_direct_reads: 0 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.create_missing_column_families: 0 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.db_log_dir: 2024-03-04T15:59:00.250 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.wal_dir: 2024-03-04T15:59:00.251 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-04T15:59:00.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-04T15:59:00.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-04T15:59:00.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-04T15:59:00.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-04T15:59:00.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-04T15:59:00.255 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.advise_random_on_open: 1 2024-03-04T15:59:00.255 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.db_write_buffer_size: 0 2024-03-04T15:59:00.255 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.write_buffer_manager: 0x55bda969d400 2024-03-04T15:59:00.255 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-04T15:59:00.255 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.rate_limiter: (nil) 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.wal_recovery_mode: 2 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enable_thread_tracking: 0 2024-03-04T15:59:00.256 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enable_pipelined_write: 0 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.unordered_write: 0 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.row_cache: None 2024-03-04T15:59:00.257 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.wal_filter: None 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_ingest_behind: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.two_write_queues: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.manual_wal_flush: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.wal_compression: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.atomic_flush: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-04T15:59:00.258 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.log_readahead_size: 0 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.best_efforts_recovery: 0 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.allow_data_in_errors: 0 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.db_host_id: __hostname__ 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_background_jobs: 2 2024-03-04T15:59:00.259 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_background_compactions: -1 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_subcompactions: 1 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_total_wal_size: 0 2024-03-04T15:59:00.260 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_open_files: -1 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bytes_per_sync: 0 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-04T15:59:00.261 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_readahead_size: 0 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_background_flushes: -1 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Compression algorithms supported: 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kZSTD supported: 0 2024-03-04T15:59:00.262 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kXpressCompression supported: 0 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kLZ4HCCompression supported: 1 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kZlibCompression supported: 1 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kSnappyCompression supported: 1 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kLZ4Compression supported: 1 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: kBZip2Compression supported: 0 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-04T15:59:00.263 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.merge_operator: 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_filter: None 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_filter_factory: None 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.sst_partitioner_factory: None 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-04T15:59:00.264 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bda96c9a00) 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: index_type: 0 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: data_block_index_type: 0 2024-03-04T15:59:00.265 INFO:journalctl@ceph.mon.c.smithi038.stdout: index_shortening: 1 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: checksum: 4 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: no_block_cache: 0 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_cache: 0x55bda96cf090 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_cache_options: 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: capacity : 536870912 2024-03-04T15:59:00.266 INFO:journalctl@ceph.mon.c.smithi038.stdout: num_shard_bits : 4 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: strict_capacity_limit : 0 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_cache_compressed: (nil) 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: persistent_cache: (nil) 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_size: 4096 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_size_deviation: 10 2024-03-04T15:59:00.267 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_restart_interval: 16 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: index_block_restart_interval: 1 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: metadata_block_size: 4096 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: partition_filters: 0 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: use_delta_encoding: 1 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: filter_policy: bloomfilter 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: whole_key_filtering: 1 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: verify_compression: 0 2024-03-04T15:59:00.268 INFO:journalctl@ceph.mon.c.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: format_version: 5 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: enable_index_compression: 1 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: block_align: 0 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: max_auto_readahead_size: 262144 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: prepopulate_block_cache: 0 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: initial_auto_readahead_size: 8192 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.write_buffer_size: 33554432 2024-03-04T15:59:00.269 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_write_buffer_number: 2 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression: NoCompression 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression: Disabled 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.prefix_extractor: nullptr 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.num_levels: 7 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-04T15:59:00.270 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.level: 32767 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.strategy: 0 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.enabled: false 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-04T15:59:00.271 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.target_file_size_base: 67108864 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-04T15:59:00.272 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.arena_block_size: 1048576 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.disable_auto_compactions: 0 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-04T15:59:00.273 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.table_properties_collectors: 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.inplace_update_support: 0 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-04T15:59:00.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.bloom_locality: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.max_successive_merges: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.paranoid_file_checks: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.force_consistency_checks: 1 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.report_bg_io_stats: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.ttl: 2592000 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enable_blob_files: false 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.min_blob_size: 0 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_file_size: 268435456 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-04T15:59:00.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.blob_file_starting_level: 0 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 03e19700-0f16-440e-947b-e47e77b75c1a 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567939822395, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567939822906, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709567939, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "03e19700-0f16-440e-947b-e47e77b75c1a", "db_session_id": "Z4XT510EBGK77A9ZI2M3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-04T15:59:00.276 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709567939822992, "job": 1, "event": "recovery_finished"} 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bda97c2000 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: DB pointer 0x55bda97ac000 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: ** DB Stats ** 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-04T15:59:00.277 INFO:journalctl@ceph.mon.c.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-04T15:59:00.278 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Cumulative compaction: 0.00 GB write, 0.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Interval compaction: 0.00 GB write, 0.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Block cache BinnedLRUCache@0x55bda96cf090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-04T15:59:00.279 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: using public_addrv v1:172.21.15.38:6790/0 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: starting mon.c rank -1 at public addrs v1:172.21.15.38:6790/0 at bind addrs v1:172.21.15.38:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(???) e0 preinit fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).mds e1 new map 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: e1 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: legacy client fscid: -1 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout: No filesystems configured 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-04T15:59:00.280 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.281 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/358238113' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:59:00.282 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='client.14184 v1:172.21.15.176:0/908284901' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Saving service mon spec with placement smithi038:[v1:172.21.15.38:6789]=a;smithi038:[v1:172.21.15.38:6790]=c;smithi176:[v1:172.21.15.176:6789]=b;count:3 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Deploying daemon mon.b on smithi176 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.283 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.a calling monitor election 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.b calling monitor election 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: monmap e2: 2 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:59:00.284 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: fsmap 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mgrmap e13: y(active, since 40s) 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: overall HEALTH_OK 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: Deploying daemon mon.c on smithi038 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/1442846988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:00.285 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:00.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 ceph-mon[38113]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-04T15:59:00.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 bash[38085]: 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 2024-03-04T15:59:00.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:58:59 smithi038 systemd[1]: Started Ceph mon.c for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:59:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-03-04T15:59:05.216 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":3,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","modified":"2024-03-04T15:58:59.849131Z","created":"2024-03-04T15:57:12.310111Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6790","nonce":0}]},"addr":"172.21.15.38:6790/0","public_addr":"172.21.15.38:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-04T15:59:05.219 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 3 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: mon.a calling monitor election 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: mon.b calling monitor election 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.220 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: mon.c calling monitor election 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: monmap e3: 3 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0,c=v1:172.21.15.38:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: fsmap 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: mgrmap e13: y(active, since 49s) 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: overall HEALTH_OK 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.221 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: mon.a calling monitor election 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: mon.b calling monitor election 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: mon.c calling monitor election 2024-03-04T15:59:05.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: monmap e3: 3 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0,c=v1:172.21.15.38:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: fsmap 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: mgrmap e13: y(active, since 49s) 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: overall HEALTH_OK 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: mon.a calling monitor election 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: mon.b calling monitor election 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: mon.c calling monitor election 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:05.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: monmap e3: 3 mons at {a=v1:172.21.15.38:6789/0,b=v1:172.21.15.176:6789/0,c=v1:172.21.15.38:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: fsmap 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: osdmap e4: 0 total, 0 up, 0 in 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: mgrmap e13: y(active, since 49s) 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: overall HEALTH_OK 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:05.705 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-04T15:59:05.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-04T15:59:06.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/625776540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:06.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/625776540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:06.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.319 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/625776540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T15:59:06.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:07.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:07.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:07.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:08 smithi038 ceph-mon[29779]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:59:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:08 smithi038 ceph-mon[29779]: Reconfiguring daemon mon.a on smithi038 2024-03-04T15:59:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:08 smithi038 ceph-mon[38113]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:59:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:08 smithi038 ceph-mon[38113]: Reconfiguring daemon mon.a on smithi038 2024-03-04T15:59:08.810 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:59:08.810 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-03-04T15:59:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T15:59:08.810 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = 172.21.15.38:6789/0 172.21.15.176:6789/0 v1:172.21.15.38:6790/0 2024-03-04T15:59:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:08 smithi176 ceph-mon[33619]: Reconfiguring mon.a (unknown last config time)... 2024-03-04T15:59:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:08 smithi176 ceph-mon[33619]: Reconfiguring daemon mon.a on smithi038 2024-03-04T15:59:09.608 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-04T15:59:09.608 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:59:09.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:59:09.633 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1014921052' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1014921052' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.634 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.635 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:09.635 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:09.635 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.638 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:59:09.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:59:09.704 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:59:09.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-03-04T15:59:09.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:09.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1014921052' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.735 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:09.735 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:09.735 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:09.735 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:09.739 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:59:09.739 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-04T15:59:09.809 INFO:tasks.cephadm:Adding mgr.y on smithi038 2024-03-04T15:59:09.809 INFO:tasks.cephadm:Adding mgr.x on smithi176 2024-03-04T15:59:09.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply mgr '2;smithi038=y;smithi176=x' 2024-03-04T15:59:10.084 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T15:59:10.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:10 smithi176 ceph-mon[33619]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:59:10.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:10 smithi176 ceph-mon[33619]: Reconfiguring daemon mon.c on smithi038 2024-03-04T15:59:10.701 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:10 smithi038 ceph-mon[38113]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:59:10.701 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:10 smithi038 ceph-mon[38113]: Reconfiguring daemon mon.c on smithi038 2024-03-04T15:59:10.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:10 smithi038 ceph-mon[29779]: Reconfiguring mon.c (monmap changed)... 2024-03-04T15:59:10.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:10 smithi038 ceph-mon[29779]: Reconfiguring daemon mon.c on smithi038 2024-03-04T15:59:11.768 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mgr update... 2024-03-04T15:59:11.768 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:11.769 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:11 smithi176 ceph-mon[33619]: Reconfiguring daemon mon.b on smithi176 2024-03-04T15:59:12.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:12.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:12.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:12.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:59:12.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:12.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:12.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:12.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[29779]: Reconfiguring daemon mon.b on smithi176 2024-03-04T15:59:12.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:12.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:12.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:12.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: Reconfiguring mon.b (monmap changed)... 2024-03-04T15:59:12.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-04T15:59:12.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-04T15:59:12.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:12.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:11 smithi038 ceph-mon[38113]: Reconfiguring daemon mon.b on smithi176 2024-03-04T15:59:13.048 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='client.24106 v1:172.21.15.176:0/108848304' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi038=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:13.048 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: Saving service mgr spec with placement smithi038=y;smithi176=x;count:2 2024-03-04T15:59:13.048 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.048 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.048 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:13.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:59:13.076 DEBUG:teuthology.orchestra.run.smithi176:mgr.x> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.x.service 2024-03-04T15:59:13.078 INFO:tasks.cephadm:Deploying OSDs... 2024-03-04T15:59:13.078 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T15:59:13.078 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:59:13.097 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:59:13.097 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-03-04T15:59:13.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='client.24106 v1:172.21.15.176:0/108848304' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi038=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: Saving service mgr spec with placement smithi038=y;smithi176=x;count:2 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='client.24106 v1:172.21.15.176:0/108848304' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi038=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:13.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: Saving service mgr spec with placement smithi038=y;smithi176=x;count:2 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:13.146 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-04 15:58:59.238267413 +0000 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-04 15:53:35.154947685 +0000 2024-03-04T15:59:13.155 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-04 15:53:35.154947685 +0000 2024-03-04T15:59:13.156 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-04 15:53:35.154947685 +0000 2024-03-04T15:59:13.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:59:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-04T15:59:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-04T15:59:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000303534 s, 1.7 MB/s 2024-03-04T15:59:13.227 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:59:13.287 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-04 15:58:59.239267394 +0000 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-04 15:53:35.510941388 +0000 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-04 15:53:35.510941388 +0000 2024-03-04T15:59:13.346 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-04 15:53:35.510941388 +0000 2024-03-04T15:59:13.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:59:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-04T15:59:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-04T15:59:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000307715 s, 1.7 MB/s 2024-03-04T15:59:13.415 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:59:13.476 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-03-04T15:59:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:59:13.534 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:13.534 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-04T15:59:13.534 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:13.535 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:13.535 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-04 15:58:59.239267394 +0000 2024-03-04T15:59:13.535 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-04 15:53:35.838935587 +0000 2024-03-04T15:59:13.535 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-04 15:53:35.838935587 +0000 2024-03-04T15:59:13.535 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-04 15:53:35.838935587 +0000 2024-03-04T15:59:13.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:59:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-04T15:59:13.603 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-04T15:59:13.604 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000276489 s, 1.9 MB/s 2024-03-04T15:59:13.605 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:59:13.662 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-04 15:58:59.240267375 +0000 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-04 15:53:36.175929626 +0000 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-04 15:53:36.175929626 +0000 2024-03-04T15:59:13.721 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-04 15:53:36.175929626 +0000 2024-03-04T15:59:13.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:59:13.792 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-04T15:59:13.792 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-04T15:59:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000273907 s, 1.9 MB/s 2024-03-04T15:59:13.794 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:59:13.852 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T15:59:13.853 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-03-04T15:59:13.871 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-04T15:59:13.871 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-04 15:58:50.069116181 +0000 2024-03-04T15:59:13.930 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-04 15:53:32.612742280 +0000 2024-03-04T15:59:13.931 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-04 15:53:32.612742280 +0000 2024-03-04T15:59:13.931 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-04 15:53:32.612742280 +0000 2024-03-04T15:59:13.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-04T15:59:13.994 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:13 smithi176 ceph-mon[33619]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:13.994 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:13.995 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:13.995 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:13 smithi176 ceph-mon[33619]: Deploying daemon mgr.x on smithi176 2024-03-04T15:59:13.997 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-04T15:59:13.997 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-04T15:59:13.997 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000203295 s, 2.5 MB/s 2024-03-04T15:59:13.998 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-04T15:59:14.055 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-04 15:58:50.069116181 +0000 2024-03-04T15:59:14.115 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-04 15:53:32.966739091 +0000 2024-03-04T15:59:14.116 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-04 15:53:32.966739091 +0000 2024-03-04T15:59:14.116 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-04 15:53:32.966739091 +0000 2024-03-04T15:59:14.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-04T15:59:14.181 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-04T15:59:14.181 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-04T15:59:14.181 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000173091 s, 3.0 MB/s 2024-03-04T15:59:14.182 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-04T15:59:14.238 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-03-04T15:59:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[29779]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[29779]: Deploying daemon mgr.x on smithi176 2024-03-04T15:59:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[38113]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:13 smithi038 ceph-mon[38113]: Deploying daemon mgr.x on smithi176 2024-03-04T15:59:14.293 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-04T15:59:14.293 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:14.293 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-04T15:59:14.293 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:14.294 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:14.294 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-04 15:58:50.070116172 +0000 2024-03-04T15:59:14.294 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-04 15:53:33.309736001 +0000 2024-03-04T15:59:14.294 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-04 15:53:33.309736001 +0000 2024-03-04T15:59:14.294 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-04 15:53:33.309736001 +0000 2024-03-04T15:59:14.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-04T15:59:14.451 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-04T15:59:14.451 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-04T15:59:14.451 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000210948 s, 2.4 MB/s 2024-03-04T15:59:14.452 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-04T15:59:14.468 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-03-04T15:59:14.529 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-03-04 15:59:14.404883857 +0000 2024-03-04T15:59:14.530 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-03-04 15:53:33.642733001 +0000 2024-03-04T15:59:14.530 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-03-04 15:53:33.642733001 +0000 2024-03-04T15:59:14.530 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-03-04 15:53:33.642733001 +0000 2024-03-04T15:59:14.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-04T15:59:14.632 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-03-04T15:59:14.632 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-03-04T15:59:14.632 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000191702 s, 2.7 MB/s 2024-03-04T15:59:14.634 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-04T15:59:14.648 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2024-03-04T15:59:14.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T15:59:14.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:14 smithi176 podman[35460]: 2024-03-04 15:59:14.839440786 +0000 UTC m=+0.085943743 container create fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:59:14.889 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:14 smithi176 podman[35460]: 2024-03-04 15:59:14.771382417 +0000 UTC m=+0.017885383 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 podman[35460]: 2024-03-04 15:59:15.008655488 +0000 UTC m=+0.255158459 container init fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 podman[35460]: 2024-03-04 15:59:15.011863844 +0000 UTC m=+0.258366802 container start fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 bash[35460]: fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 2024-03-04T15:59:15.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 systemd[1]: Started Ceph mgr.x for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:15.573+0000 7f7e7adfd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:15.669+0000 7f7e7adfd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:15.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:15.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:16.186 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:16.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:16.354+0000 7f7e7adfd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T15:59:16.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:16.447+0000 7f7e7adfd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T15:59:16.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:16.598+0000 7f7e7adfd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T15:59:16.882 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T15:59:16.883 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T15:59:16.883 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-04T15:59:16.883 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-04T15:59:16.883 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0100829 s, 1.0 GB/s 2024-03-04T15:59:16.883 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:16.695+0000 7f7e7adfd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:16.780+0000 7f7e7adfd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:17.044+0000 7f7e7adfd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.046 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.047 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:16 smithi176 ceph-mon[33619]: Reconfiguring daemon mgr.y on smithi038 2024-03-04T15:59:17.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:17.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[29779]: Reconfiguring daemon mgr.y on smithi038 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.217 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: Reconfiguring mgr.y (unknown last config time)... 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-04T15:59:17.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:17.219 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:16 smithi038 ceph-mon[38113]: Reconfiguring daemon mgr.y on smithi038 2024-03-04T15:59:17.337 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:17.154+0000 7f7e7adfd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T15:59:17.337 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:17.241+0000 7f7e7adfd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T15:59:17.337 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:17.335+0000 7f7e7adfd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T15:59:17.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-03-04T15:59:17.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:17.511+0000 7f7e7adfd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T15:59:18.185 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:59:18.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:18.472+0000 7f7e7adfd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T15:59:18.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:18.601+0000 7f7e7adfd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T15:59:19.219 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:18 smithi176 ceph-mon[33619]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:19.219 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:18.935+0000 7f7e7adfd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T15:59:19.219 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:19.024+0000 7f7e7adfd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T15:59:19.219 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:19.217+0000 7f7e7adfd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:59:19.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:18 smithi038 ceph-mon[29779]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:19.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:18 smithi038 ceph-mon[38113]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:19.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:19.400+0000 7f7e7adfd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T15:59:19.943 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:19.682+0000 7f7e7adfd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T15:59:19.943 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:19.941+0000 7f7e7adfd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T15:59:20.241 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:20.177+0000 7f7e7adfd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T15:59:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:20 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:20.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:20 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:20 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:20.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:20 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:21.251 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:20.980+0000 7f7e7adfd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='client.34115 v1:172.21.15.38:0/259390540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='client.34115 v1:172.21.15.38:0/259390540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:21.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:21 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='client.34115 v1:172.21.15.38:0/259390540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:21.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:21.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:21.550+0000 7f7e7adfd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T15:59:21.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:21.648+0000 7f7e7adfd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T15:59:22.272 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 15:59:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T15:59:21.863+0000 7f7e7adfd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[29779]: Standby manager daemon x started 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:59:22.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[38113]: Standby manager daemon x started 2024-03-04T15:59:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:59:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:59:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:59:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:22 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:59:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:22 smithi176 ceph-mon[33619]: Standby manager daemon x started 2024-03-04T15:59:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:22 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T15:59:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:22 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T15:59:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:22 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T15:59:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:22 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/2691986407' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2418374424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]': finished 2024-03-04T15:59:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:59:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:23 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2418374424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]': finished 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:59:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: mgrmap e14: y(active, since 66s), standbys: x 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2418374424' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: from='client.34121 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]: dispatch 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: from='client.34121 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244"}]': finished 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: osdmap e5: 1 total, 0 up, 1 in 2024-03-04T15:59:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:23 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1447113332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1447113332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1447113332' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:25.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:25 smithi176 ceph-mon[33619]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:25 smithi038 ceph-mon[29779]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:25.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:25 smithi038 ceph-mon[38113]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:27 smithi176 ceph-mon[33619]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:27 smithi038 ceph-mon[29779]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:27 smithi038 ceph-mon[38113]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:29.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[29779]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:29.548 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:59:29.548 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:29.548 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[38113]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:29.548 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:59:29.548 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:29 smithi176 ceph-mon[33619]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-04T15:59:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:30 smithi038 ceph-mon[38113]: Deploying daemon osd.0 on smithi038 2024-03-04T15:59:30.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:30 smithi038 ceph-mon[29779]: Deploying daemon osd.0 on smithi038 2024-03-04T15:59:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:30 smithi176 ceph-mon[33619]: Deploying daemon osd.0 on smithi038 2024-03-04T15:59:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:31 smithi038 ceph-mon[29779]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:31 smithi038 ceph-mon[38113]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:31.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:31 smithi176 ceph-mon[33619]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:33.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:33 smithi038 ceph-mon[29779]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:33.605 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:33 smithi038 ceph-mon[38113]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:33 smithi176 ceph-mon[33619]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[29779]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[38113]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:35.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:35 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:36.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:35 smithi176 ceph-mon[33619]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:36.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:35 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:36.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:35 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:36.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:35 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:37.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:38.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:38.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:38.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.447 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-03-04T15:59:38.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[29779]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[29779]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[38113]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[38113]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:39.057 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.0.service 2024-03-04T15:59:39.059 INFO:tasks.cephadm:Deploying osd.1 on smithi038 with /dev/vg_nvme/lv_3... 2024-03-04T15:59:39.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T15:59:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:38 smithi176 ceph-mon[33619]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:39.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:38 smithi176 ceph-mon[33619]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-04T15:59:39.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:39.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:39.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:39 smithi176 ceph-mon[33619]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.115 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[29779]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: osdmap e6: 1 total, 0 up, 1 in 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: from='osd.0 v1:172.21.15.38:6801/1080993770' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.116 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-mon[38113]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T15:59:40.116 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0[42115]: 2024-03-04T15:59:39.758+0000 7f6098bdb700 -1 osd.0 0 waiting for initial osdmap 2024-03-04T15:59:40.116 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 15:59:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0[42115]: 2024-03-04T15:59:39.760+0000 7f60949ce700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T15:59:40.159 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:59:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[38113]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T15:59:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[38113]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:59:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[29779]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T15:59:40.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[29779]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:59:40.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:40.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:40 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:40 smithi176 ceph-mon[33619]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:40 smithi176 ceph-mon[33619]: osdmap e7: 1 total, 0 up, 1 in 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:40 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:40 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: osd.0 v1:172.21.15.38:6801/1080993770 boot 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:41.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:41.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: osd.0 v1:172.21.15.38:6801/1080993770 boot 2024-03-04T15:59:41.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:59:41.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:41 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: osd.0 v1:172.21.15.38:6801/1080993770 boot 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: osdmap e8: 1 total, 1 up, 1 in 2024-03-04T15:59:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:41 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T15:59:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T15:59:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T15:59:42.460 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-04T15:59:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-04T15:59:42.461 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.016511 s, 635 MB/s 2024-03-04T15:59:42.461 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-04T15:59:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: osdmap e9: 1 total, 1 up, 1 in 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T15:59:43.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T15:59:43.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-03-04T15:59:43.543 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T15:59:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[29779]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi038 2024-03-04T15:59:43.984 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-04T15:59:43.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[38113]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:43.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi038 2024-03-04T15:59:43.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:43 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-04T15:59:44.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:43 smithi176 ceph-mon[33619]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:44.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:43 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi038 2024-03-04T15:59:44.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:43 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 16069M 2024-03-04T15:59:46.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[29779]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[38113]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:46.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:45 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:45 smithi176 ceph-mon[33619]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:45 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T15:59:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:45 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T15:59:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:45 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:46.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:46 smithi038 ceph-mon[38113]: from='client.34142 v1:172.21.15.38:0/2744932963' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:46.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:46 smithi038 ceph-mon[29779]: from='client.34142 v1:172.21.15.38:0/2744932963' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:47.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:46 smithi176 ceph-mon[33619]: from='client.34142 v1:172.21.15.38:0/2744932963' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T15:59:48.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:47 smithi176 ceph-mon[33619]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:47 smithi038 ceph-mon[29779]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:48.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:47 smithi038 ceph-mon[38113]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4033223277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]': finished 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:59:49.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3496076920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4033223277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]': finished 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3496076920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4033223277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: from='client.34148 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]: dispatch 2024-03-04T15:59:49.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: from='client.34148 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51"}]': finished 2024-03-04T15:59:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: osdmap e10: 2 total, 1 up, 2 in 2024-03-04T15:59:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T15:59:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3496076920' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T15:59:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:49 smithi176 ceph-mon[33619]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:50.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:49 smithi038 ceph-mon[29779]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:49 smithi038 ceph-mon[38113]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:51 smithi038 ceph-mon[29779]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:51 smithi038 ceph-mon[38113]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:51 smithi176 ceph-mon[33619]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:54.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:53 smithi038 ceph-mon[29779]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:54.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:53 smithi038 ceph-mon[38113]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:53 smithi176 ceph-mon[33619]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[29779]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:59:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:55.212 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[29779]: Deploying daemon osd.1 on smithi038 2024-03-04T15:59:55.213 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[38113]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:55.213 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:59:55.213 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:55.213 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:54 smithi038 ceph-mon[38113]: Deploying daemon osd.1 on smithi038 2024-03-04T15:59:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:54 smithi176 ceph-mon[33619]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:54 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-04T15:59:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:54 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T15:59:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:54 smithi176 ceph-mon[33619]: Deploying daemon osd.1 on smithi038 2024-03-04T15:59:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:57 smithi038 ceph-mon[29779]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:57 smithi038 ceph-mon[38113]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:57.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:57 smithi176 ceph-mon[33619]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:59.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 15:59:59 smithi038 ceph-mon[29779]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:59.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 15:59:59 smithi038 ceph-mon[38113]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T15:59:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 15:59:59 smithi176 ceph-mon[33619]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[29779]: overall HEALTH_OK 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:00 smithi038 ceph-mon[38113]: overall HEALTH_OK 2024-03-04T16:00:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:00 smithi176 ceph-mon[33619]: overall HEALTH_OK 2024-03-04T16:00:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:01 smithi038 ceph-mon[29779]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:01 smithi038 ceph-mon[38113]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:01 smithi176 ceph-mon[33619]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:03.397 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 1 on host 'smithi038' 2024-03-04T16:00:03.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.678 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.679 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:03.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:03.950 DEBUG:teuthology.orchestra.run.smithi038:osd.1> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.1.service 2024-03-04T16:00:03.952 INFO:tasks.cephadm:Deploying osd.2 on smithi038 with /dev/vg_nvme/lv_2... 2024-03-04T16:00:03.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T16:00:04.616 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[29779]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[29779]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[29779]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[29779]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:05.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[38113]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:00:05.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[38113]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T16:00:05.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:05.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[38113]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:05.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-mon[38113]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:05.068 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1[47947]: 2024-03-04T16:00:04.769+0000 7f042f2a8700 -1 osd.1 0 waiting for initial osdmap 2024-03-04T16:00:05.068 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1[47947]: 2024-03-04T16:00:04.772+0000 7f042b09b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:04 smithi176 ceph-mon[33619]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-04T16:00:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:04 smithi176 ceph-mon[33619]: osdmap e11: 2 total, 1 up, 2 in 2024-03-04T16:00:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:04 smithi176 ceph-mon[33619]: from='osd.1 v1:172.21.15.38:6805/2331598100' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:04 smithi176 ceph-mon[33619]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:07.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:07.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T16:00:07.958 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.959 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.959 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:06 smithi176 ceph-mon[33619]: osd.1 v1:172.21.15.38:6805/2331598100 boot 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:06 smithi176 ceph-mon[33619]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T16:00:07.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:06 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: osdmap e12: 2 total, 1 up, 2 in 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:00:07.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[29779]: osd.1 v1:172.21.15.38:6805/2331598100 boot 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[29779]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[38113]: osd.1 v1:172.21.15.38:6805/2331598100 boot 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[38113]: osdmap e13: 2 total, 2 up, 2 in 2024-03-04T16:00:07.962 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:06 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:00:07.962 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T16:00:07.962 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T16:00:07.963 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-04T16:00:07.963 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-04T16:00:07.963 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0162383 s, 646 MB/s 2024-03-04T16:00:07.963 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-04T16:00:08.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:07 smithi038 ceph-mon[29779]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:08.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:07 smithi038 ceph-mon[29779]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T16:00:08.014 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:07 smithi038 ceph-mon[38113]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:08.014 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:07 smithi038 ceph-mon[38113]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T16:00:08.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-03-04T16:00:08.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:07 smithi176 ceph-mon[33619]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-04T16:00:08.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:07 smithi176 ceph-mon[33619]: osdmap e14: 2 total, 2 up, 2 in 2024-03-04T16:00:08.527 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:09.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:09.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi038 2024-03-04T16:00:09.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 8034M 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:09.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi038 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 8034M 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:09.534 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:09 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi038 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 8034M 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:09.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:09.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:09 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:11.097 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[38113]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[38113]: from='client.14274 v1:172.21.15.38:0/381743339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[29779]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[29779]: from='client.14274 v1:172.21.15.38:0/381743339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:11.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:11.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:11 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:11 smithi176 ceph-mon[33619]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:11 smithi176 ceph-mon[33619]: from='client.14274 v1:172.21.15.38:0/381743339' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:11 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1379648921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]': finished 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T16:00:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1379648921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]': finished 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T16:00:13.395 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1379648921' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: from='client.34175 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]: dispatch 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: from='client.34175 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3"}]': finished 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: osdmap e15: 3 total, 2 up, 3 in 2024-03-04T16:00:13.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:14.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1338089181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:14.416 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1338089181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:14.416 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1338089181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:15.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:15 smithi176 ceph-mon[33619]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:15.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:15 smithi038 ceph-mon[29779]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:15.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:15 smithi038 ceph-mon[38113]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:17.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:17 smithi038 ceph-mon[29779]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:17.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:17 smithi038 ceph-mon[38113]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:17.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:17 smithi176 ceph-mon[33619]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:19 smithi176 ceph-mon[33619]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:19.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:19 smithi038 ceph-mon[29779]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:19.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:19 smithi038 ceph-mon[38113]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:00:20.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:20.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[29779]: Deploying daemon osd.2 on smithi038 2024-03-04T16:00:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:00:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:20 smithi038 ceph-mon[38113]: Deploying daemon osd.2 on smithi038 2024-03-04T16:00:20.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:20 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-04T16:00:20.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:20 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:20.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:20 smithi176 ceph-mon[33619]: Deploying daemon osd.2 on smithi038 2024-03-04T16:00:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:21 smithi176 ceph-mon[33619]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:21.404 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:21 smithi038 ceph-mon[29779]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:21.404 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:21 smithi038 ceph-mon[38113]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:23.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:23 smithi176 ceph-mon[33619]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:23.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:23 smithi038 ceph-mon[29779]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:23.419 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:23 smithi038 ceph-mon[38113]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[38113]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[29779]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:25 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:25 smithi176 ceph-mon[33619]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:25 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:25 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:25 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:27.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[29779]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:27.908 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:27.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[38113]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:27.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:27.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:27 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:27.911 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:27 smithi176 ceph-mon[33619]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:27.911 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:27 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:27.911 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:27 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:28.250 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 2 on host 'smithi038' 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[29779]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[38113]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:28.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:28 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:28.807 DEBUG:teuthology.orchestra.run.smithi038:osd.2> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.2.service 2024-03-04T16:00:28.809 INFO:tasks.cephadm:Deploying osd.3 on smithi038 with /dev/vg_nvme/lv_1... 2024-03-04T16:00:28.809 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T16:00:29.104 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:28 smithi176 ceph-mon[33619]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-04T16:00:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:28 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:28 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:29.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[29779]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:29.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[29779]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:00:29.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[29779]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T16:00:29.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[29779]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[38113]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[38113]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[38113]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[38113]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:29.877 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:29.877 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2[53987]: 2024-03-04T16:00:29.675+0000 7f3f25c44700 -1 osd.2 0 waiting for initial osdmap 2024-03-04T16:00:29.878 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:29 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2[53987]: 2024-03-04T16:00:29.681+0000 7f3f21a37700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:30.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:29 smithi176 ceph-mon[33619]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:30.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:29 smithi176 ceph-mon[33619]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-04T16:00:30.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:29 smithi176 ceph-mon[33619]: osdmap e16: 3 total, 2 up, 3 in 2024-03-04T16:00:30.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:29 smithi176 ceph-mon[33619]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:30.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:30.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[29779]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:30.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[29779]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T16:00:30.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:30.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:30.824 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[38113]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:30.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[38113]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T16:00:30.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:30.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116027 s, 904 MB/s 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 2024-03-04T16:00:30.965 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-04T16:00:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:30 smithi176 ceph-mon[33619]: from='osd.2 v1:172.21.15.38:6809/1898914368' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:31.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:30 smithi176 ceph-mon[33619]: osdmap e17: 3 total, 2 up, 3 in 2024-03-04T16:00:31.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.780 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: osd.2 v1:172.21.15.38:6809/1898914368 boot 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T16:00:31.781 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: osd.2 v1:172.21.15.38:6809/1898914368 boot 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T16:00:31.782 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:31 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:31.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: osd.2 v1:172.21.15.38:6809/1898914368 boot 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: osdmap e18: 3 total, 3 up, 3 in 2024-03-04T16:00:32.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:31 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:00:32.609 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:00:32.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:32 smithi038 ceph-mon[29779]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T16:00:32.860 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:32.860 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:32 smithi038 ceph-mon[38113]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T16:00:32.860 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:32 smithi176 ceph-mon[33619]: osdmap e19: 3 total, 3 up, 3 in 2024-03-04T16:00:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[29779]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[29779]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[38113]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[38113]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T16:00:33.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:33 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:33 smithi176 ceph-mon[33619]: pgmap v77: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:34.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:33 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:34.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:33 smithi176 ceph-mon[33619]: osdmap e20: 3 total, 3 up, 3 in 2024-03-04T16:00:34.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:33 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:00:34.199 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58047]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:34.200 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58047]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:34.200 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58047]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:34.200 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58047]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:34.482 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58084]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:34.482 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58084]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:34.482 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58084]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:34.482 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58084]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:34.741 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58126]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:34.742 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58126]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:34.742 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58126]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:34.742 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58126]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:34.993 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58150]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:34.994 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58150]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:34.994 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58150]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:34.994 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:00:34 smithi038 sudo[58150]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:34.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:34.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T16:00:34.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:34.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:34.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:34 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:00:35.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: osdmap e21: 3 total, 3 up, 3 in 2024-03-04T16:00:35.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:35.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:34 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:35.482 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58219]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:35.482 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:35.482 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58219]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:35.482 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58219]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:35.743 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58278]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:35.743 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58278]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:35.744 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58278]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:35.744 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58278]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:36.015 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58286]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:36.016 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58286]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:36.016 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58286]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:36.016 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:35 smithi038 sudo[58286]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi038 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 5356M 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:36.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:36.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:36.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi038 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 5356M 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:36.444 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58302]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58302]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58302]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58302]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi038 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 5356M 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: osdmap e22: 3 total, 3 up, 3 in 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:36.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:36.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36187]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:36.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36187]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:36.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36187]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:36.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36187]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36190]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36190]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36190]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:36 smithi176 sudo[36190]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:36.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58335]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-04T16:00:36.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58335]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58335]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:36 smithi038 sudo[58335]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='client.34202 v1:172.21.15.38:0/2892995824' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:37.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:37.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:37 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 sudo[58345]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-04T16:00:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 sudo[58345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-04T16:00:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 sudo[58345]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 sudo[58345]: pam_unix(sudo:session): session closed for user root 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='client.34202 v1:172.21.15.38:0/2892995824' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='client.34202 v1:172.21.15.38:0/2892995824' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:37.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:00:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:00:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:37 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1112029872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]': finished 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T16:00:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1112029872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]': finished 2024-03-04T16:00:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1112029872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='client.34211 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]: dispatch 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='client.34211 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4"}]': finished 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: osdmap e23: 4 total, 3 up, 4 in 2024-03-04T16:00:38.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:39.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:39 smithi176 ceph-mon[33619]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:39.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:39 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2726579792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:39 smithi038 ceph-mon[29779]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:39 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2726579792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:39 smithi038 ceph-mon[38113]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:39 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2726579792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:00:41.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:41 smithi176 ceph-mon[33619]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:41 smithi038 ceph-mon[29779]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:41 smithi038 ceph-mon[38113]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:43.400 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:43 smithi038 ceph-mon[29779]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 61 KiB/s wr, 5 op/s 2024-03-04T16:00:43.400 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:43 smithi038 ceph-mon[38113]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 61 KiB/s wr, 5 op/s 2024-03-04T16:00:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:43 smithi176 ceph-mon[33619]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 61 KiB/s wr, 5 op/s 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[38113]: Deploying daemon osd.3 on smithi038 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:44.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:44 smithi038 ceph-mon[29779]: Deploying daemon osd.3 on smithi038 2024-03-04T16:00:44.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-04T16:00:44.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:44.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:44 smithi176 ceph-mon[33619]: Deploying daemon osd.3 on smithi038 2024-03-04T16:00:45.341 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:45 smithi038 ceph-mon[29779]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 42 KiB/s wr, 3 op/s 2024-03-04T16:00:45.341 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:45 smithi038 ceph-mon[38113]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 42 KiB/s wr, 3 op/s 2024-03-04T16:00:45.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:45 smithi176 ceph-mon[33619]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 42 KiB/s wr, 3 op/s 2024-03-04T16:00:47.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:47 smithi038 ceph-mon[29779]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-03-04T16:00:47.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:47 smithi038 ceph-mon[38113]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-03-04T16:00:47.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:47 smithi176 ceph-mon[33619]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 32 KiB/s wr, 2 op/s 2024-03-04T16:00:49.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[29779]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 894 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[38113]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 894 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:49 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:49 smithi176 ceph-mon[33619]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 894 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-04T16:00:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:49 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:49 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:49 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:51.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:51 smithi038 ceph-mon[29779]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:51.418 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:51 smithi038 ceph-mon[38113]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:51 smithi176 ceph-mon[33619]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:52.599 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 3 on host 'smithi038' 2024-03-04T16:00:53.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:53.218 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.243 DEBUG:teuthology.orchestra.run.smithi038:osd.3> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.3.service 2024-03-04T16:00:53.245 INFO:tasks.cephadm:Deploying osd.4 on smithi176 with /dev/vg_nvme/lv_4... 2024-03-04T16:00:53.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-04T16:00:53.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-04T16:00:53.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:53.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:00:53.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:53.491 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.106 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:00:54.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: osdmap e24: 4 total, 3 up, 4 in 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: from='osd.3 v1:172.21.15.38:6813/552035408' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:00:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:00:54.680 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:00:54.421+0000 7ff1c912a700 -1 osd.3 0 waiting for initial osdmap 2024-03-04T16:00:54.680 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:00:54.425+0000 7ff1c0f15700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:00:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[29779]: from='osd.3 ' entity='osd.3' 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:54 smithi038 ceph-mon[38113]: from='osd.3 ' entity='osd.3' 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: osdmap e25: 4 total, 3 up, 4 in 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:55.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:54 smithi176 ceph-mon[33619]: from='osd.3 ' entity='osd.3' 2024-03-04T16:00:55.367 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-04T16:00:55.368 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-04T16:00:55.368 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-04T16:00:55.368 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-04T16:00:55.368 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113231 s, 926 MB/s 2024-03-04T16:00:55.368 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-04T16:00:55.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_4 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: osd.3 v1:172.21.15.38:6813/552035408 boot 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:56.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:00:56.141 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:00:56.141 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:56.141 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: osd.3 v1:172.21.15.38:6813/552035408 boot 2024-03-04T16:00:56.141 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T16:00:56.141 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:56.256 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:00:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:00:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:00:56.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:56.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: osd.3 v1:172.21.15.38:6813/552035408 boot 2024-03-04T16:00:56.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: osdmap e26: 4 total, 4 up, 4 in 2024-03-04T16:00:56.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi038 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 4017M 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.493 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:57 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi038 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 4017M 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi038 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 4017M 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:00:57.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:57 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[29779]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[29779]: from='client.24235 v1:172.21.15.176:0/2382211233' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[38113]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[38113]: from='client.24235 v1:172.21.15.176:0/2382211233' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:58 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:58 smithi176 ceph-mon[33619]: osdmap e27: 4 total, 4 up, 4 in 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:58 smithi176 ceph-mon[33619]: from='client.24235 v1:172.21.15.176:0/2382211233' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:58 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:58 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:00:58.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:58 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:00:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:00:59 smithi038 ceph-mon[29779]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:00:59.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:00:59 smithi038 ceph-mon[38113]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:00:59.501 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:00:59 smithi176 ceph-mon[33619]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:01.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:01.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/221445356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]': finished 2024-03-04T16:01:01.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T16:01:01.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:01.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2782707441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/221445356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]': finished 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2782707441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/221445356' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]: dispatch 2024-03-04T16:01:01.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3"}]': finished 2024-03-04T16:01:01.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: osdmap e28: 5 total, 4 up, 5 in 2024-03-04T16:01:01.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:01.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2782707441' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:03.306 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:03 smithi176 ceph-mon[33619]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:03 smithi038 ceph-mon[29779]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:03 smithi038 ceph-mon[38113]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:05.299 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:05 smithi176 ceph-mon[33619]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:05 smithi038 ceph-mon[29779]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:05 smithi038 ceph-mon[38113]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:06.185 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:01:06.186 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:01:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:01:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:01:06.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:01:06.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:01:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T16:01:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:01:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:07 smithi176 ceph-mon[33619]: Deploying daemon osd.4 on smithi176 2024-03-04T16:01:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[29779]: Deploying daemon osd.4 on smithi176 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: osdmap e29: 5 total, 4 up, 5 in 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:07 smithi038 ceph-mon[38113]: Deploying daemon osd.4 on smithi176 2024-03-04T16:01:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:09 smithi176 ceph-mon[33619]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:09.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:09 smithi038 ceph-mon[29779]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:09.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:09 smithi038 ceph-mon[38113]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:11.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:11 smithi176 ceph-mon[33619]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:11 smithi038 ceph-mon[29779]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:11 smithi038 ceph-mon[38113]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:12 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:12.542 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.542 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:12.542 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:12 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:13.472 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:13 smithi176 ceph-mon[33619]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:13 smithi038 ceph-mon[29779]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:13 smithi038 ceph-mon[38113]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:14.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:14.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:14.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:14.512 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 4 on host 'smithi176' 2024-03-04T16:01:15.171 DEBUG:teuthology.orchestra.run.smithi176:osd.4> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.4.service 2024-03-04T16:01:15.174 INFO:tasks.cephadm:Deploying osd.5 on smithi176 with /dev/vg_nvme/lv_3... 2024-03-04T16:01:15.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-04T16:01:15.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:15.430 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.430 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.430 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:15.430 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:15.430 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:15 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:15.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:15.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:15.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:15 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:16.254 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:01:16.441 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-mon[33619]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:01:16.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-mon[33619]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T16:01:16.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:16.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-mon[33619]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:16.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-mon[33619]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:16.442 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:01:16.249+0000 7f5919862700 -1 osd.4 0 waiting for initial osdmap 2024-03-04T16:01:16.442 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:01:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:01:16.254+0000 7f5915655700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:01:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[29779]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[29779]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[29779]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[29779]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[38113]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[38113]: osdmap e30: 5 total, 4 up, 5 in 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[38113]: from='osd.4 v1:172.21.15.176:6800/3997489718' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:16 smithi038 ceph-mon[38113]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:17 smithi176 ceph-mon[33619]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:17.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:17 smithi176 ceph-mon[33619]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:17.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:17 smithi176 ceph-mon[33619]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T16:01:17.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:17 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:17.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:17 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[29779]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[29779]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[29779]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[38113]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[38113]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[38113]: osdmap e31: 5 total, 4 up, 5 in 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:17 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:18.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: osd.4 v1:172.21.15.176:6800/3997489718 boot 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:18.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:18 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: osd.4 v1:172.21.15.176:6800/3997489718 boot 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:18.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:01:18.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:01:18.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: osd.4 v1:172.21.15.176:6800/3997489718 boot 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: osdmap e32: 5 total, 5 up, 5 in 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:18.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:18 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:19.102 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-04T16:01:19.102 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-04T16:01:19.102 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-04T16:01:19.102 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-04T16:01:19.103 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150567 s, 696 MB/s 2024-03-04T16:01:19.103 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-04T16:01:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:19 smithi176 ceph-mon[33619]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:19 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi176 2024-03-04T16:01:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:19 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 17088M 2024-03-04T16:01:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:19 smithi176 ceph-mon[33619]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T16:01:19.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_3 2024-03-04T16:01:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[38113]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi176 2024-03-04T16:01:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 17088M 2024-03-04T16:01:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[38113]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T16:01:19.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[29779]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-04T16:01:19.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi176 2024-03-04T16:01:19.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 17088M 2024-03-04T16:01:19.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:19 smithi038 ceph-mon[29779]: osdmap e33: 5 total, 5 up, 5 in 2024-03-04T16:01:19.876 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:01:20.511 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:20 smithi176 ceph-mon[33619]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T16:01:20.511 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:20 smithi176 ceph-mon[33619]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T16:01:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:20 smithi038 ceph-mon[29779]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T16:01:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:20 smithi038 ceph-mon[29779]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T16:01:20.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:20 smithi038 ceph-mon[38113]: osdmap e34: 5 total, 5 up, 5 in 2024-03-04T16:01:20.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:20 smithi038 ceph-mon[38113]: osdmap e35: 5 total, 5 up, 5 in 2024-03-04T16:01:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:21 smithi176 ceph-mon[33619]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:21.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:21 smithi176 ceph-mon[33619]: osdmap e36: 5 total, 5 up, 5 in 2024-03-04T16:01:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:21 smithi038 ceph-mon[29779]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:21.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:21 smithi038 ceph-mon[29779]: osdmap e36: 5 total, 5 up, 5 in 2024-03-04T16:01:21.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:21 smithi038 ceph-mon[38113]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:21.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:21 smithi038 ceph-mon[38113]: osdmap e36: 5 total, 5 up, 5 in 2024-03-04T16:01:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:22 smithi176 ceph-mon[33619]: from='client.24259 v1:172.21.15.176:0/2311183845' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:22 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:22 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:22 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[29779]: from='client.24259 v1:172.21.15.176:0/2311183845' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[38113]: from='client.24259 v1:172.21.15.176:0/2311183845' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:22 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:23 smithi176 ceph-mon[33619]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:01:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:23 smithi038 ceph-mon[29779]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:01:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:23 smithi038 ceph-mon[38113]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:01:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/577159563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[38113]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[38113]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]': finished 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[38113]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/577159563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[29779]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[29779]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]': finished 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[29779]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:01:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:24 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/577159563' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:24 smithi176 ceph-mon[33619]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]: dispatch 2024-03-04T16:01:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:24 smithi176 ceph-mon[33619]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06"}]': finished 2024-03-04T16:01:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:24 smithi176 ceph-mon[33619]: osdmap e37: 6 total, 5 up, 6 in 2024-03-04T16:01:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:24 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:25.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:25 smithi176 ceph-mon[33619]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T16:01:25.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/307750879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:25.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:25 smithi038 ceph-mon[29779]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T16:01:25.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/307750879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:25.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:25 smithi038 ceph-mon[38113]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-04T16:01:25.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/307750879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:27 smithi038 ceph-mon[29779]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:27 smithi038 ceph-mon[38113]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:27 smithi176 ceph-mon[33619]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:29 smithi176 ceph-mon[33619]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:29 smithi038 ceph-mon[29779]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:29 smithi038 ceph-mon[38113]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:01:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:30.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:01:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:30.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-04T16:01:30.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:31.474 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:31 smithi176 ceph-mon[33619]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:01:31.475 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:31 smithi176 ceph-mon[33619]: Deploying daemon osd.5 on smithi176 2024-03-04T16:01:31.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:31 smithi038 ceph-mon[29779]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:01:31.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:31 smithi038 ceph-mon[29779]: Deploying daemon osd.5 on smithi176 2024-03-04T16:01:31.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:31 smithi038 ceph-mon[38113]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-04T16:01:31.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:31 smithi038 ceph-mon[38113]: Deploying daemon osd.5 on smithi176 2024-03-04T16:01:33.526 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:33 smithi176 ceph-mon[33619]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:33 smithi038 ceph-mon[29779]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:33.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:33 smithi038 ceph-mon[38113]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:34.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:34 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:01:34.730 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:34 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:01:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:34 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:01:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:34 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:01:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:34 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:01:34.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:34 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:01:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[29779]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[29779]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:01:35.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:35.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[38113]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:35.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[38113]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:01:35.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:35 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:35 smithi176 ceph-mon[33619]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:35 smithi176 ceph-mon[33619]: osdmap e38: 6 total, 5 up, 6 in 2024-03-04T16:01:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:35 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:36.823 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:36 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:36 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:37 smithi176 ceph-mon[33619]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:37 smithi038 ceph-mon[29779]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:37.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:37 smithi038 ceph-mon[38113]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:38.875 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:38 smithi176 ceph-mon[33619]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:38.931 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 5 on host 'smithi176' 2024-03-04T16:01:38.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[29779]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:38.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:38.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:38.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:38 smithi038 ceph-mon[38113]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-04T16:01:39.465 DEBUG:teuthology.orchestra.run.smithi176:osd.5> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.5.service 2024-03-04T16:01:39.468 INFO:tasks.cephadm:Deploying osd.6 on smithi176 with /dev/vg_nvme/lv_2... 2024-03-04T16:01:39.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-04T16:01:39.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: osdmap e39: 6 total, 5 up, 6 in 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:39.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: osdmap e39: 6 total, 5 up, 6 in 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:39.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:40.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: osdmap e39: 6 total, 5 up, 6 in 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='osd.5 v1:172.21.15.176:6804/1267814386' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:40.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:01:40.396 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:01:40.118+0000 7f2315940700 -1 osd.5 0 waiting for initial osdmap 2024-03-04T16:01:40.397 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:01:40.125+0000 7f230d72b700 -1 osd.5 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:01:40.481 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:01:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: osdmap e40: 6 total, 5 up, 6 in 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: osdmap e41: 6 total, 5 up, 6 in 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[29779]: from='osd.5 ' entity='osd.5' 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: osdmap e40: 6 total, 5 up, 6 in 2024-03-04T16:01:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: osdmap e41: 6 total, 5 up, 6 in 2024-03-04T16:01:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:40.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:40 smithi038 ceph-mon[38113]: from='osd.5 ' entity='osd.5' 2024-03-04T16:01:41.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: osdmap e40: 6 total, 5 up, 6 in 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: osdmap e41: 6 total, 5 up, 6 in 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:40 smithi176 ceph-mon[33619]: from='osd.5 ' entity='osd.5' 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: osd.5 v1:172.21.15.176:6804/1267814386 boot 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:01:41.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:41 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: osd.5 v1:172.21.15.176:6804/1267814386 boot 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-04T16:01:41.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:41.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: osd.5 v1:172.21.15.176:6804/1267814386 boot 2024-03-04T16:01:41.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: osdmap e42: 6 total, 6 up, 6 in 2024-03-04T16:01:41.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:41 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:01:42.647 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-04T16:01:42.647 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-04T16:01:42.647 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-04T16:01:42.647 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-04T16:01:42.648 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012729 s, 824 MB/s 2024-03-04T16:01:42.648 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-04T16:01:43.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:43 smithi176 ceph-mon[33619]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:43.285 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:43 smithi176 ceph-mon[33619]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:01:43.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_2 2024-03-04T16:01:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:43 smithi038 ceph-mon[29779]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:43 smithi038 ceph-mon[29779]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:01:43.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:43 smithi038 ceph-mon[38113]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:43.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:43 smithi038 ceph-mon[38113]: osdmap e43: 6 total, 6 up, 6 in 2024-03-04T16:01:43.762 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: osdmap e44: 6 total, 6 up, 6 in 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi176 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 8544M 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.199 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:44.200 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:44.200 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: osdmap e44: 6 total, 6 up, 6 in 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi176 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 8544M 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: osdmap e44: 6 total, 6 up, 6 in 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi176 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 8544M 2024-03-04T16:01:44.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:44.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:44.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:01:44.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:01:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:45 smithi176 ceph-mon[33619]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:45 smithi176 ceph-mon[33619]: osdmap e45: 6 total, 6 up, 6 in 2024-03-04T16:01:45.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:45 smithi038 ceph-mon[29779]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:45.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:45 smithi038 ceph-mon[29779]: osdmap e45: 6 total, 6 up, 6 in 2024-03-04T16:01:45.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:45 smithi038 ceph-mon[38113]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:45.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:45 smithi038 ceph-mon[38113]: osdmap e45: 6 total, 6 up, 6 in 2024-03-04T16:01:46.224 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:46 smithi176 ceph-mon[33619]: osdmap e46: 6 total, 6 up, 6 in 2024-03-04T16:01:46.225 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:46 smithi176 ceph-mon[33619]: from='client.24286 v1:172.21.15.176:0/1563943683' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:46.225 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:46 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:46.225 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:46 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:46.225 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:46 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[29779]: osdmap e46: 6 total, 6 up, 6 in 2024-03-04T16:01:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[29779]: from='client.24286 v1:172.21.15.176:0/1563943683' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[38113]: osdmap e46: 6 total, 6 up, 6 in 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[38113]: from='client.24286 v1:172.21.15.176:0/1563943683' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:01:46.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:46 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:47.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:47 smithi038 ceph-mon[29779]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:47.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:47 smithi038 ceph-mon[38113]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:47 smithi176 ceph-mon[33619]: pgmap v141: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2758261818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[29779]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[29779]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]': finished 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[29779]: osdmap e47: 7 total, 6 up, 7 in 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2758261818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[38113]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[38113]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]': finished 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[38113]: osdmap e47: 7 total, 6 up, 7 in 2024-03-04T16:01:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:48 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2758261818' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:48 smithi176 ceph-mon[33619]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]: dispatch 2024-03-04T16:01:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:48 smithi176 ceph-mon[33619]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4"}]': finished 2024-03-04T16:01:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:48 smithi176 ceph-mon[33619]: osdmap e47: 7 total, 6 up, 7 in 2024-03-04T16:01:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:48 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:01:49.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:49 smithi038 ceph-mon[29779]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:01:49.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/154815266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:49.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:49 smithi038 ceph-mon[38113]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:01:49.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/154815266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:49 smithi176 ceph-mon[33619]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:01:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/154815266' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:01:51.452 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:51 smithi176 ceph-mon[33619]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:51 smithi038 ceph-mon[29779]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:51 smithi038 ceph-mon[38113]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:01:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:53 smithi176 ceph-mon[33619]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:53 smithi038 ceph-mon[29779]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:53.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:53 smithi038 ceph-mon[38113]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-04T16:01:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[29779]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-04T16:01:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[29779]: Deploying daemon osd.6 on smithi176 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[38113]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:55.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:55 smithi038 ceph-mon[38113]: Deploying daemon osd.6 on smithi176 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:55 smithi176 ceph-mon[33619]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-04T16:01:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:01:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:55 smithi176 ceph-mon[33619]: Deploying daemon osd.6 on smithi176 2024-03-04T16:01:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:01:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[29779]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:57 smithi038 ceph-mon[38113]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:57 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:01:57.494 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:57 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:01:57.495 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:57 smithi176 ceph-mon[33619]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:01:59.292 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:01:59 smithi176 ceph-mon[33619]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:01:59 smithi038 ceph-mon[29779]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:01:59.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:01:59 smithi038 ceph-mon[38113]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:00 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:00 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:01.736 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:01 smithi176 ceph-mon[33619]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:01 smithi038 ceph-mon[29779]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:01 smithi038 ceph-mon[38113]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:03.263 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 6 on host 'smithi176' 2024-03-04T16:02:03.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.517 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:03.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:03.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:03.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:03.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:03.807 DEBUG:teuthology.orchestra.run.smithi176:osd.6> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.6.service 2024-03-04T16:02:03.809 INFO:tasks.cephadm:Deploying osd.7 on smithi176 with /dev/vg_nvme/lv_1... 2024-03-04T16:02:03.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-04T16:02:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-mon[33619]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:02:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-mon[33619]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-mon[33619]: osdmap e48: 7 total, 6 up, 7 in 2024-03-04T16:02:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-mon[33619]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[29779]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:02:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[29779]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[29779]: osdmap e48: 7 total, 6 up, 7 in 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[29779]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[38113]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[38113]: from='osd.6 v1:172.21.15.176:6808/587062426' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[38113]: osdmap e48: 7 total, 6 up, 7 in 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:04 smithi038 ceph-mon[38113]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:05.028 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:05.249 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:02:04 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:02:04.996+0000 7f6b5d584700 -1 osd.6 0 waiting for initial osdmap 2024-03-04T16:02:05.250 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:02:05.004+0000 7f6b54b6e700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: osdmap e49: 7 total, 6 up, 7 in 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: from='osd.6 ' entity='osd.6' 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: osd.6 v1:172.21.15.176:6808/587062426 boot 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:02:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:05 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: osdmap e49: 7 total, 6 up, 7 in 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: from='osd.6 ' entity='osd.6' 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: osd.6 v1:172.21.15.176:6808/587062426 boot 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-04T16:02:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: osdmap e49: 7 total, 6 up, 7 in 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: from='osd.6 ' entity='osd.6' 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: osd.6 v1:172.21.15.176:6808/587062426 boot 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: osdmap e50: 7 total, 7 up, 7 in 2024-03-04T16:02:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:05 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:02:06.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:06.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:06.836 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:06 smithi176 ceph-mon[33619]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:02:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:06.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:06.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[29779]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:02:06.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:06.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:06.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:06 smithi038 ceph-mon[38113]: osdmap e51: 7 total, 7 up, 7 in 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125222 s, 837 MB/s 2024-03-04T16:02:07.102 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-03-04T16:02:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:07 smithi176 ceph-mon[33619]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:07 smithi176 ceph-mon[33619]: osdmap e52: 7 total, 7 up, 7 in 2024-03-04T16:02:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:07 smithi038 ceph-mon[29779]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:07 smithi038 ceph-mon[29779]: osdmap e52: 7 total, 7 up, 7 in 2024-03-04T16:02:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:07 smithi038 ceph-mon[38113]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:07 smithi038 ceph-mon[38113]: osdmap e52: 7 total, 7 up, 7 in 2024-03-04T16:02:08.348 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi176:vg_nvme/lv_1 2024-03-04T16:02:08.590 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:09.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:09.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi176 2024-03-04T16:02:09.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 5696M 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[29779]: osdmap e53: 7 total, 7 up, 7 in 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi176 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 5696M 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:09.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:09.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:08 smithi038 ceph-mon[38113]: osdmap e53: 7 total, 7 up, 7 in 2024-03-04T16:02:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: pgmap v158: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi176 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 5696M 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:09.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:08 smithi176 ceph-mon[33619]: osdmap e53: 7 total, 7 up, 7 in 2024-03-04T16:02:10.895 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:10 smithi176 ceph-mon[33619]: osdmap e54: 7 total, 7 up, 7 in 2024-03-04T16:02:10.895 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:10 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:02:10.895 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:10 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:02:10.895 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:10 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[29779]: osdmap e54: 7 total, 7 up, 7 in 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[38113]: osdmap e54: 7 total, 7 up, 7 in 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-04T16:02:10.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:10 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:11.898 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:11 smithi176 ceph-mon[33619]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:11.898 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:11 smithi176 ceph-mon[33619]: from='client.24313 v1:172.21.15.176:0/659332108' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:11 smithi038 ceph-mon[29779]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:11 smithi038 ceph-mon[29779]: from='client.24313 v1:172.21.15.176:0/659332108' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:11 smithi038 ceph-mon[38113]: pgmap v161: 1 pgs: 1 remapped+peering; 577 KiB data, 417 MiB used, 625 GiB / 626 GiB avail 2024-03-04T16:02:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:11 smithi038 ceph-mon[38113]: from='client.24313 v1:172.21.15.176:0/659332108' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:02:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2052325964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]': finished 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: osdmap e55: 8 total, 7 up, 8 in 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2870462220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2052325964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]': finished 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: osdmap e55: 8 total, 7 up, 8 in 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:14.002 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2870462220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:02:14.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-04T16:02:14.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2052325964' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]: dispatch 2024-03-04T16:02:14.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e"}]': finished 2024-03-04T16:02:14.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: osdmap e55: 8 total, 7 up, 8 in 2024-03-04T16:02:14.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:14.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2870462220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-04T16:02:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:15 smithi038 ceph-mon[29779]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:15 smithi038 ceph-mon[38113]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:16.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:15 smithi176 ceph-mon[33619]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:17 smithi176 ceph-mon[33619]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:02:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:17 smithi038 ceph-mon[29779]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:02:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:17 smithi038 ceph-mon[38113]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-04T16:02:19.974 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:19 smithi176 ceph-mon[33619]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T16:02:19.974 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:19 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:02:19.975 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:19 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[29779]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T16:02:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:02:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:19.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[38113]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-03-04T16:02:19.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-04T16:02:19.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:19 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:20.702 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:20 smithi176 ceph-mon[33619]: Deploying daemon osd.7 on smithi176 2024-03-04T16:02:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:20 smithi038 ceph-mon[29779]: Deploying daemon osd.7 on smithi176 2024-03-04T16:02:20.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:20 smithi038 ceph-mon[38113]: Deploying daemon osd.7 on smithi176 2024-03-04T16:02:21.900 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:21 smithi176 ceph-mon[33619]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:02:21.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:21 smithi038 ceph-mon[29779]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:02:21.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:21 smithi038 ceph-mon[38113]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-04T16:02:23.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:23 smithi038 ceph-mon[38113]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:23.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:23 smithi038 ceph-mon[29779]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:24.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:23 smithi176 ceph-mon[33619]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: osdmap e56: 8 total, 7 up, 8 in 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: osdmap e56: 8 total, 7 up, 8 in 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:24.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:24 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: osdmap e56: 8 total, 7 up, 8 in 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:25.071 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:24 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:25 smithi038 ceph-mon[38113]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:25 smithi038 ceph-mon[29779]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:25 smithi176 ceph-mon[33619]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:27 smithi176 ceph-mon[33619]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:27 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:27 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:27 smithi176 ceph-mon[33619]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:27 smithi176 ceph-mon[33619]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.196 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 7 on host 'smithi176' 2024-03-04T16:02:28.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[38113]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[38113]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[38113]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[29779]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[29779]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:27 smithi038 ceph-mon[29779]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-04T16:02:28.817 DEBUG:teuthology.orchestra.run.smithi176:osd.7> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.7.service 2024-03-04T16:02:28.820 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-04T16:02:28.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-04T16:02:29.077 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: osdmap e57: 8 total, 7 up, 8 in 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.369 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-mon[33619]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.432 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: osdmap e57: 8 total, 7 up, 8 in 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:29.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[29779]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: osdmap e57: 8 total, 7 up, 8 in 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:29.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='osd.7 v1:172.21.15.176:6812/1019151537' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.435 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:29 smithi038 ceph-mon[38113]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-03-04T16:02:29.853 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:02:29.844+0000 7f49158b8700 -1 osd.7 0 waiting for initial osdmap 2024-03-04T16:02:29.853 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:02:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:02:29.852+0000 7f490d6a3700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: osdmap e58: 8 total, 7 up, 8 in 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: from='osd.7 ' entity='osd.7' 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: osd.7 v1:172.21.15.176:6812/1019151537 boot 2024-03-04T16:02:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: osdmap e58: 8 total, 7 up, 8 in 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: from='osd.7 ' entity='osd.7' 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: osd.7 v1:172.21.15.176:6812/1019151537 boot 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:02:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:30 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: osdmap e58: 8 total, 7 up, 8 in 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: from='osd.7 ' entity='osd.7' 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: osd.7 v1:172.21.15.176:6812/1019151537 boot 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: osdmap e59: 8 total, 8 up, 8 in 2024-03-04T16:02:30.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:30 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:02:31.462 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1709568150,"num_in_osds":8,"osd_in_since":1709568132,"num_remapped_pgs":1} 2024-03-04T16:02:31.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[29779]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1928267669' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[29779]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:02:31.475 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:31.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:31.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[38113]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:31.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1928267669' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:02:31.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:31 smithi038 ceph-mon[38113]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:02:31.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:31 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:31.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:31 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:31.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:31 smithi176 ceph-mon[33619]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-04T16:02:31.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:31 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1928267669' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-04T16:02:31.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:31 smithi176 ceph-mon[33619]: osdmap e60: 8 total, 8 up, 8 in 2024-03-04T16:02:31.716 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: Detected new or changed devices on smithi176 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.900 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 4272M 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[29779]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:32.901 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: Detected new or changed devices on smithi176 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 4272M 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:32.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:32.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:32.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:32.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:32 smithi038 ceph-mon[38113]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: Detected new or changed devices on smithi176 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 4272M 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:33.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:33.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:33.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:32 smithi176 ceph-mon[33619]: osdmap e61: 8 total, 8 up, 8 in 2024-03-04T16:02:33.439 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:02:33.439 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":62,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:57:15.255572+0000","modified":"2024-03-04T16:02:33.068019+0000","last_up_change":"2024-03-04T16:02:30.058990+0000","last_in_change":"2024-03-04T16:02:12.888408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:00:32.201931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"99ecf38d-7bc9-412d-a194-f3d9eed8e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6801","nonce":1080993770}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6802","nonce":1080993770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6804","nonce":1080993770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6803","nonce":1080993770}]},"public_addr":"172.21.15.38:6801/1080993770","cluster_addr":"172.21.15.38:6802/1080993770","heartbeat_back_addr":"172.21.15.38:6804/1080993770","heartbeat_front_addr":"172.21.15.38:6803/1080993770","state":["exists","up"]},{"osd":1,"uuid":"f8c66eb4-4966-45ce-af41-b6d01713eb51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6805","nonce":2331598100}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6806","nonce":2331598100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6808","nonce":2331598100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6807","nonce":2331598100}]},"public_addr":"172.21.15.38:6805/2331598100","cluster_addr":"172.21.15.38:6806/2331598100","heartbeat_back_addr":"172.21.15.38:6808/2331598100","heartbeat_front_addr":"172.21.15.38:6807/2331598100","state":["exists","up"]},{"osd":2,"uuid":"afb2dd7e-d27a-4536-bb80-e5724b39ffd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6809","nonce":1898914368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6810","nonce":1898914368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6812","nonce":1898914368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6811","nonce":1898914368}]},"public_addr":"172.21.15.38:6809/1898914368","cluster_addr":"172.21.15.38:6810/1898914368","heartbeat_back_addr":"172.21.15.38:6812/1898914368","heartbeat_front_addr":"172.21.15.38:6811/1898914368","state":["exists","up"]},{"osd":3,"uuid":"13f08b82-22d5-4466-9ad4-27ffa15ed7c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6813","nonce":552035408}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6814","nonce":552035408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6816","nonce":552035408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6815","nonce":552035408}]},"public_addr":"172.21.15.38:6813/552035408","cluster_addr":"172.21.15.38:6814/552035408","heartbeat_back_addr":"172.21.15.38:6816/552035408","heartbeat_front_addr":"172.21.15.38:6815/552035408","state":["exists","up"]},{"osd":4,"uuid":"a58d697b-e1de-44cc-b791-8a8d68fd01f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6800","nonce":3997489718}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6801","nonce":3997489718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6803","nonce":3997489718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6802","nonce":3997489718}]},"public_addr":"172.21.15.176:6800/3997489718","cluster_addr":"172.21.15.176:6801/3997489718","heartbeat_back_addr":"172.21.15.176:6803/3997489718","heartbeat_front_addr":"172.21.15.176:6802/3997489718","state":["exists","up"]},{"osd":5,"uuid":"7adb17cc-4843-401e-a7fd-b726c4a99a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6804","nonce":1267814386}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6805","nonce":1267814386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6807","nonce":1267814386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6806","nonce":1267814386}]},"public_addr":"172.21.15.176:6804/1267814386","cluster_addr":"172.21.15.176:6805/1267814386","heartbeat_back_addr":"172.21.15.176:6807/1267814386","heartbeat_front_addr":"172.21.15.176:6806/1267814386","state":["exists","up"]},{"osd":6,"uuid":"0e12d14b-c02d-4acf-9784-8680f8ee40f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6808","nonce":587062426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6809","nonce":587062426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6811","nonce":587062426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6810","nonce":587062426}]},"public_addr":"172.21.15.176:6808/587062426","cluster_addr":"172.21.15.176:6809/587062426","heartbeat_back_addr":"172.21.15.176:6811/587062426","heartbeat_front_addr":"172.21.15.176:6810/587062426","state":["exists","up"]},{"osd":7,"uuid":"1f348507-1649-4e25-b5f5-cb67c6cd045e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6812","nonce":1019151537}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6813","nonce":1019151537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6815","nonce":1019151537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6814","nonce":1019151537}]},"public_addr":"172.21.15.176:6812/1019151537","cluster_addr":"172.21.15.176:6813/1019151537","heartbeat_back_addr":"172.21.15.176:6815/1019151537","heartbeat_front_addr":"172.21.15.176:6814/1019151537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:23.477799+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:01:05.197928+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:01:33.523659+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:02.988045+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:01:15.275891+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:01:39.650607+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:03.678248+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:28.516206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/175394725":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/1249306019":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/2156780629":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/1997350354":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2476964715":"2024-03-05T15:58:15.846725+0000","172.21.15.38:0/1178105870":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/3751720132":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2774521881":"2024-03-05T15:58:15.846725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:02:34.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[29779]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:34.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[29779]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:34.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3826364872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:02:34.013 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[38113]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:34.013 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[38113]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:34.014 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3826364872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:02:34.045 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-04T16:02:34.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-04T16:02:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:33 smithi176 ceph-mon[33619]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:34.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:33 smithi176 ceph-mon[33619]: osdmap e62: 8 total, 8 up, 8 in 2024-03-04T16:02:34.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3826364872' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:02:34.331 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:02:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:35.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:35.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:35.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:35.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:35.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 476 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:35.425 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:35.426 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: osdmap e63: 8 total, 8 up, 8 in 2024-03-04T16:02:35.426 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:35.426 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:36.124 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-04T16:02:36.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:36 smithi038 ceph-mon[29779]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:36.454 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:36 smithi038 ceph-mon[38113]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:36.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:36 smithi176 ceph-mon[33619]: osdmap e64: 8 total, 8 up, 8 in 2024-03-04T16:02:36.708 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi038 2024-03-04T16:02:36.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi038=foo.a' 2024-03-04T16:02:36.961 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:37.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:37 smithi176 ceph-mon[33619]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:02:37.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3453001029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:02:37.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:37 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:02:37.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:37 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:02:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[38113]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3453001029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[29779]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3453001029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:02:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:37 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:02:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:38 smithi038 ceph-mon[29779]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:38 smithi038 ceph-mon[38113]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:38.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:38 smithi176 ceph-mon[33619]: osdmap e65: 8 total, 8 up, 8 in 2024-03-04T16:02:38.617 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled rgw.foo.a update... 2024-03-04T16:02:39.154 DEBUG:teuthology.orchestra.run.smithi038:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@rgw.foo.a.service 2024-03-04T16:02:39.157 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi176 2024-03-04T16:02:39.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-04T16:02:39.398 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:39.813 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='client.24355 v1:172.21.15.176:0/1617972066' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:39.814 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:39.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:39 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:39.910 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='client.24355 v1:172.21.15.176:0/1617972066' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:39.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='client.24355 v1:172.21.15.176:0/1617972066' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi038=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:39.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:39 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:40.818 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:40 smithi176 ceph-mon[33619]: Deploying daemon rgw.foo.a on smithi038 2024-03-04T16:02:40.912 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:40 smithi038 ceph-mon[29779]: Deploying daemon rgw.foo.a on smithi038 2024-03-04T16:02:40.913 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:40 smithi038 ceph-mon[38113]: Deploying daemon rgw.foo.a on smithi038 2024-03-04T16:02:41.494 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 systemd[1]: Starting Ceph rgw.foo.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:41.660 INFO:teuthology.orchestra.run.smithi176.stderr:pool 'datapool' created 2024-03-04T16:02:41.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[29779]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:41.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2774133119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:41.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[29779]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:41.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[38113]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:41.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2774133119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:41.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:41 smithi038 ceph-mon[38113]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:41.844 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 podman[65194]: 2024-03-04 16:02:41.491619992 +0000 UTC m=+0.018338583 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:02:41.844 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 podman[65194]: 2024-03-04 16:02:41.598594082 +0000 UTC m=+0.125312662 container create cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-04T16:02:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:41 smithi176 ceph-mon[33619]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-04T16:02:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2774133119' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:42.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:41 smithi176 ceph-mon[33619]: from='client.24361 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-04T16:02:42.202 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 podman[65194]: 2024-03-04 16:02:41.843141661 +0000 UTC m=+0.369860239 container init cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:02:42.203 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 podman[65194]: 2024-03-04 16:02:41.845713825 +0000 UTC m=+0.372432402 container start cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:02:42.203 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 bash[65194]: cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b 2024-03-04T16:02:42.203 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:02:41 smithi038 systemd[1]: Started Ceph rgw.foo.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:42.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- rbd pool init datapool 2024-03-04T16:02:42.657 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:42.975 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:42 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-04T16:02:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='client.24361 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: osdmap e66: 8 total, 8 up, 8 in 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:42.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:42 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:43.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[29779]: pgmap v189: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-04T16:02:43.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[29779]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:43.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:02:43.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[29779]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:43.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:43.764 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[38113]: pgmap v189: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-04T16:02:43.764 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[38113]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:43.764 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:02:43.764 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[38113]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:43.764 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:43 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:43 smithi176 ceph-mon[33619]: pgmap v189: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-04T16:02:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:43 smithi176 ceph-mon[33619]: Saving service rgw.foo.a spec with placement smithi038=foo.a;count:1 2024-03-04T16:02:43.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:43 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:02:43.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:43 smithi176 ceph-mon[33619]: osdmap e67: 8 total, 8 up, 8 in 2024-03-04T16:02:43.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:43 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-04T16:02:44.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:44.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/2824882265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[29779]: from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:44.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:44.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:44.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/2824882265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:44.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:44 smithi038 ceph-mon[38113]: from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2735438447' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: osdmap e68: 8 total, 8 up, 8 in 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/2824882265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:44 smithi176 ceph-mon[33619]: from='client.24385 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: pgmap v192: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: Checking dashboard <-> RGW credentials 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:45.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: pgmap v192: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: Checking dashboard <-> RGW credentials 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:45.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:45 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: pgmap v192: 36 pgs: 5 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:02:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: Checking dashboard <-> RGW credentials 2024-03-04T16:02:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: from='client.24385 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-04T16:02:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: osdmap e69: 8 total, 8 up, 8 in 2024-03-04T16:02:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:46.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:45 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-04T16:02:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[29779]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[38113]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:46 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:47.052 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:47.052 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:46 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-04T16:02:47.052 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:46 smithi176 ceph-mon[33619]: osdmap e70: 8 total, 8 up, 8 in 2024-03-04T16:02:47.052 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:46 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:47.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.176 --placement '1;smithi176=iscsi.a' 2024-03-04T16:02:47.525 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:47.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: pgmap v195: 68 pgs: 10 active+clean, 8 creating+peering, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T16:02:47.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:02:47.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:02:47.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:47.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: pgmap v195: 68 pgs: 10 active+clean, 8 creating+peering, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T16:02:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:02:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:02:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: pgmap v195: 68 pgs: 10 active+clean, 8 creating+peering, 50 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: osdmap e71: 8 total, 8 up, 8 in 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-04T16:02:48.961 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:48 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:48.962 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:48.962 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:48 smithi176 ceph-mon[33619]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:49.211 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled iscsi.datapool update... 2024-03-04T16:02:49.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:49.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:49.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[29779]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-04T16:02:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:48 smithi038 ceph-mon[38113]: osdmap e72: 8 total, 8 up, 8 in 2024-03-04T16:02:49.733 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-04T16:02:49.733 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T16:02:49.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-04T16:02:49.771 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T16:02:49.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-04T16:02:49.809 DEBUG:teuthology.orchestra.run.smithi176:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@iscsi.iscsi.a.service 2024-03-04T16:02:49.852 INFO:tasks.cephadm:Adding prometheus.a on smithi176 2024-03-04T16:02:49.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply prometheus '1;smithi176=a' 2024-03-04T16:02:50.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: pgmap v198: 100 pgs: 52 active+clean, 14 creating+peering, 34 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T16:02:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:02:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:49 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:50.126 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:50.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: pgmap v198: 100 pgs: 52 active+clean, 14 creating+peering, 34 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T16:02:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:02:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:50.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: pgmap v198: 100 pgs: 52 active+clean, 14 creating+peering, 34 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-04T16:02:50.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: osdmap e73: 8 total, 8 up, 8 in 2024-03-04T16:02:50.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-04T16:02:50.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:49 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.24421 v1:172.21.15.176:0/1891956260' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:02:51.112 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:50 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.24421 v1:172.21.15.176:0/1891956260' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-03-04T16:02:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.24421 v1:172.21.15.176:0/1891956260' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.176", "placement": "1;smithi176=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: Saving service iscsi.datapool spec with placement smithi176=iscsi.a;count:1 2024-03-04T16:02:51.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-04T16:02:51.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: osdmap e74: 8 total, 8 up, 8 in 2024-03-04T16:02:51.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3397116038' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:50 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-04T16:02:51.930 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled prometheus update... 2024-03-04T16:02:52.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: pgmap v201: 132 pgs: 69 active+clean, 10 creating+peering, 53 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-04T16:02:52.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:02:52.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:51 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:52.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: pgmap v201: 132 pgs: 69 active+clean, 10 creating+peering, 53 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-04T16:02:52.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: pgmap v201: 132 pgs: 69 active+clean, 10 creating+peering, 53 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 2 op/s 2024-03-04T16:02:52.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2081174426' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='client.34406 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: osdmap e75: 8 total, 8 up, 8 in 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-04T16:02:52.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:51 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:02:52.927 DEBUG:teuthology.orchestra.run.smithi176:prometheus.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@prometheus.a.service 2024-03-04T16:02:52.929 INFO:tasks.cephadm:Adding node-exporter.a on smithi038 2024-03-04T16:02:52.930 INFO:tasks.cephadm:Adding node-exporter.b on smithi176 2024-03-04T16:02:52.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply node-exporter '2;smithi038=a;smithi176=b' 2024-03-04T16:02:52.952 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:52 smithi176 ceph-mon[33619]: Deploying daemon iscsi.iscsi.a on smithi176 2024-03-04T16:02:52.952 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:52 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:52 smithi038 ceph-mon[29779]: Deploying daemon iscsi.iscsi.a on smithi176 2024-03-04T16:02:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:52 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:52 smithi038 ceph-mon[38113]: Deploying daemon iscsi.iscsi.a on smithi176 2024-03-04T16:02:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:52 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:53.368 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:54.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:53 smithi176 ceph-mon[33619]: pgmap v203: 132 pgs: 127 active+clean, 5 unknown; 581 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.0 KiB/s wr, 143 op/s 2024-03-04T16:02:54.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:53 smithi176 ceph-mon[33619]: from='client.24442 v1:172.21.15.176:0/4183569926' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:54.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:53 smithi176 ceph-mon[33619]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-04T16:02:54.109 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:53 smithi176 systemd[1]: Starting Ceph iscsi.iscsi.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:02:54.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[29779]: pgmap v203: 132 pgs: 127 active+clean, 5 unknown; 581 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.0 KiB/s wr, 143 op/s 2024-03-04T16:02:54.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[29779]: from='client.24442 v1:172.21.15.176:0/4183569926' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:54.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[29779]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-04T16:02:54.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[38113]: pgmap v203: 132 pgs: 127 active+clean, 5 unknown; 581 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 6.0 KiB/s wr, 143 op/s 2024-03-04T16:02:54.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[38113]: from='client.24442 v1:172.21.15.176:0/4183569926' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:54.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:53 smithi038 ceph-mon[38113]: Saving service prometheus spec with placement smithi176=a;count:1 2024-03-04T16:02:54.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 podman[60503]: 2024-03-04 16:02:54.130842602 +0000 UTC m=+0.019725217 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:02:54.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 podman[60503]: 2024-03-04 16:02:54.468893323 +0000 UTC m=+0.357775964 container create 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux ) 2024-03-04T16:02:54.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 podman[60503]: 2024-03-04 16:02:54.763334844 +0000 UTC m=+0.652217459 container init 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, ceph=True) 2024-03-04T16:02:54.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 podman[60503]: 2024-03-04 16:02:54.766680668 +0000 UTC m=+0.655563267 container start 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=) 2024-03-04T16:02:55.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 bash[60503]: 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 2024-03-04T16:02:55.090 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:54 smithi176 systemd[1]: Started Ceph iscsi.iscsi.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:02:55.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Started the configuration object watcher 2024-03-04T16:02:55.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Checking for config object changes every 1s 2024-03-04T16:02:55.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Processing osd blocklist entries for this node 2024-03-04T16:02:55.565 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled node-exporter update... 2024-03-04T16:02:55.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Reading the configuration object to update local LIO configuration 2024-03-04T16:02:55.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Configuration does not have an entry for this host(smithi176.front.sepia.ceph.com) - nothing to define to LIO 2024-03-04T16:02:55.678 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-04T16:02:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: pgmap v204: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.5 KiB/s wr, 145 op/s 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/302374275' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:02:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:55 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: pgmap v204: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.5 KiB/s wr, 145 op/s 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/302374275' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: pgmap v204: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 5.5 KiB/s wr, 145 op/s 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: osdmap e76: 8 total, 8 up, 8 in 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/302374275' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-04T16:02:56.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:55 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:56.517 DEBUG:teuthology.orchestra.run.smithi038:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.a.service 2024-03-04T16:02:56.520 DEBUG:teuthology.orchestra.run.smithi176:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.b.service 2024-03-04T16:02:56.523 INFO:tasks.cephadm:Adding alertmanager.a on smithi038 2024-03-04T16:02:56.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply alertmanager '1;smithi038=a' 2024-03-04T16:02:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:56 smithi176 ceph-mon[33619]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-04T16:02:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:56 smithi176 ceph-mon[33619]: from='client.24478 v1:172.21.15.176:0/819028884' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi038=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:56 smithi176 ceph-mon[33619]: Saving service node-exporter spec with placement smithi038=a;smithi176=b;count:2 2024-03-04T16:02:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:56 smithi176 ceph-mon[33619]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:02:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:56 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[29779]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-04T16:02:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[29779]: from='client.24478 v1:172.21.15.176:0/819028884' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi038=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[29779]: Saving service node-exporter spec with placement smithi038=a;smithi176=b;count:2 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[29779]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[38113]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[38113]: from='client.24478 v1:172.21.15.176:0/819028884' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi038=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[38113]: Saving service node-exporter spec with placement smithi038=a;smithi176=b;count:2 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[38113]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-04T16:02:57.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:56 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:02:57.375 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:02:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:57 smithi176 ceph-mon[33619]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.0 KiB/s wr, 164 op/s 2024-03-04T16:02:58.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:57 smithi038 ceph-mon[38113]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.0 KiB/s wr, 164 op/s 2024-03-04T16:02:58.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:57 smithi038 ceph-mon[29779]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 6.0 KiB/s wr, 164 op/s 2024-03-04T16:02:59.113 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled alertmanager update... 2024-03-04T16:03:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-04T16:03:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:02:59 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.182 DEBUG:teuthology.orchestra.run.smithi038:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@alertmanager.a.service 2024-03-04T16:03:00.185 INFO:tasks.cephadm:Adding grafana.a on smithi176 2024-03-04T16:03:00.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph orch apply grafana '1;smithi176=a' 2024-03-04T16:03:00.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-04T16:03:00.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.7 KiB/s wr, 142 op/s 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:00.207 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:00.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:02:59 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:00.429 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:03:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:00 smithi176 ceph-mon[33619]: from='client.24484 v1:172.21.15.176:0/2088406601' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:00 smithi176 ceph-mon[33619]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-04T16:03:01.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:00 smithi038 ceph-mon[38113]: from='client.24484 v1:172.21.15.176:0/2088406601' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:01.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:00 smithi038 ceph-mon[38113]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-04T16:03:01.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:00 smithi038 ceph-mon[29779]: from='client.24484 v1:172.21.15.176:0/2088406601' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi038=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:01.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:00 smithi038 ceph-mon[29779]: Saving service alertmanager spec with placement smithi038=a;count:1 2024-03-04T16:03:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:01 smithi176 ceph-mon[33619]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-04T16:03:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:01 smithi176 ceph-mon[33619]: Deploying daemon prometheus.a on smithi176 2024-03-04T16:03:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:01 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:02.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:02.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:02.201 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled grafana update... 2024-03-04T16:03:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[29779]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-04T16:03:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[29779]: Deploying daemon prometheus.a on smithi176 2024-03-04T16:03:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[38113]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 4.1 KiB/s wr, 124 op/s 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[38113]: Deploying daemon prometheus.a on smithi176 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:03.076 DEBUG:teuthology.orchestra.run.smithi176:grafana.a> sudo journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@grafana.a.service 2024-03-04T16:03:03.078 INFO:tasks.cephadm:Setting up client nodes... 2024-03-04T16:03:03.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T16:03:03.334 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:03.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:03 smithi176 ceph-mon[33619]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.3 KiB/s wr, 54 op/s 2024-03-04T16:03:03.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:03 smithi176 ceph-mon[33619]: from='client.24490 v1:172.21.15.176:0/478775006' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:03.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:03 smithi176 ceph-mon[33619]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-04T16:03:03.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:03 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:03.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:03 smithi176 ceph-mon[33619]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:03.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[29779]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.3 KiB/s wr, 54 op/s 2024-03-04T16:03:03.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[29779]: from='client.24490 v1:172.21.15.176:0/478775006' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:03.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[29779]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-04T16:03:03.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:03.457 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[29779]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:03.457 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[38113]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 1.3 KiB/s wr, 54 op/s 2024-03-04T16:03:03.457 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[38113]: from='client.24490 v1:172.21.15.176:0/478775006' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:03:03.457 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[38113]: Saving service grafana spec with placement smithi176=a;count:1 2024-03-04T16:03:03.458 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:03.458 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:03 smithi038 ceph-mon[38113]: osdmap e77: 8 total, 8 up, 8 in 2024-03-04T16:03:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:05 smithi176 ceph-mon[33619]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 579 B/s wr, 30 op/s 2024-03-04T16:03:05.359 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:03:05 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:03:05.411 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:05 smithi038 ceph-mon[29779]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 579 B/s wr, 30 op/s 2024-03-04T16:03:05.411 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:05 smithi038 ceph-mon[38113]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 579 B/s wr, 30 op/s 2024-03-04T16:03:05.484 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-03-04T16:03:05.485 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQC58OVlHIo7HBAA4Q/NNE5+Ukj/8SwieIR67w== 2024-03-04T16:03:06.139 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-04T16:03:06.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-04T16:03:06.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-04T16:03:06.181 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-04T16:03:06.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:06.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/619481163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[29779]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[29779]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:06.386 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:06.386 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/619481163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.386 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[38113]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.386 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:06 smithi038 ceph-mon[38113]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:06 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/619481163' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:06 smithi176 ceph-mon[33619]: from='client.34481 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:06 smithi176 ceph-mon[33619]: from='client.34481 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:07.149 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:03:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:07 smithi038 ceph-mon[29779]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-04T16:03:07.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:07 smithi038 ceph-mon[38113]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-04T16:03:07.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:07 smithi176 ceph-mon[33619]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 6.7 KiB/s rd, 102 B/s wr, 13 op/s 2024-03-04T16:03:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:09 smithi176 ceph-mon[33619]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:09 smithi038 ceph-mon[29779]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:09 smithi038 ceph-mon[38113]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:11.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:11 smithi176 ceph-mon[33619]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:11 smithi038 ceph-mon[29779]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:11 smithi038 ceph-mon[38113]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:12.069 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 podman[61997]: 2024-03-04 16:03:11.936437466 +0000 UTC m=+0.028989552 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-04T16:03:12.069 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 podman[61997]: 2024-03-04 16:03:12.068495895 +0000 UTC m=+0.161047955 container create 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:03:12.137 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-03-04T16:03:12.138 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDA8OVlfvDJBxAA0jMvc2Sa8FuvTtkkp1fK0A== 2024-03-04T16:03:12.690 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-03-04T16:03:12.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-04T16:03:12.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-04T16:03:12.715 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.176:0/8854701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.715 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-mon[33619]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.715 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-mon[33619]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:12.733 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-04T16:03:12.733 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-04T16:03:12.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-04T16:03:12.992 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.176:0/8854701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[29779]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[29779]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.176:0/8854701' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[38113]: from='client.24502 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-04T16:03:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:12 smithi038 ceph-mon[38113]: from='client.24502 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 podman[61997]: 2024-03-04 16:03:12.816099938 +0000 UTC m=+0.908652004 container init 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 podman[61997]: 2024-03-04 16:03:12.818982253 +0000 UTC m=+0.911534314 container start 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.842Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.842Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.842Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi176 (none))" 2024-03-04T16:03:13.109 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.842Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.842Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.843Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.844Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.846Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.846Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.847Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.847Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.659µs 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.847Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.848Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.848Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.383µs wal_replay_duration=329.435µs wbl_replay_duration=122ns total_replay_duration=374.783µs 2024-03-04T16:03:13.110 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.848Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.848Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.848Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.869Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.529991ms db_storage=837ns remote_storage=1.237µs web_handler=301ns query_engine=646ns scrape=411.635µs scrape_sd=78.367µs notify=945ns notify_sd=2.34µs rules=19.836621ms tracing=6.325µs 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.869Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:03:12.869Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 bash[61997]: 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 2024-03-04T16:03:13.111 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:03:12 smithi176 systemd[1]: Started Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:03:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[38113]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[29779]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:13 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:03:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:13 smithi176 ceph-mon[33619]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' 2024-03-04T16:03:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:13 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-04T16:03:14.355 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setuser ceph since I am not root 2024-03-04T16:03:14.355 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ignoring --setgroup ceph since I am not root 2024-03-04T16:03:14.358 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: ignoring --setuser ceph since I am not root 2024-03-04T16:03:14.359 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: ignoring --setgroup ceph since I am not root 2024-03-04T16:03:14.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[29779]: 1.0 deep-scrub starts 2024-03-04T16:03:14.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[29779]: 1.0 deep-scrub ok 2024-03-04T16:03:14.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[29779]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:03:14.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[29779]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-04T16:03:14.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[38113]: 1.0 deep-scrub starts 2024-03-04T16:03:14.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[38113]: 1.0 deep-scrub ok 2024-03-04T16:03:14.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[38113]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:03:14.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-mon[38113]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-04T16:03:14.617 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:14.614+0000 7fcecb4be200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:03:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-mon[33619]: 1.0 deep-scrub starts 2024-03-04T16:03:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-mon[33619]: 1.0 deep-scrub ok 2024-03-04T16:03:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-mon[33619]: from='mgr.14152 v1:172.21.15.38:0/2984363' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-04T16:03:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-mon[33619]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-04T16:03:14.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:14.585+0000 7f150d4ea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-04T16:03:14.859 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:14.680+0000 7f150d4ea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:03:14.993 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:14.714+0000 7fcecb4be200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-04T16:03:15.018 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:03:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2897073311' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:03:15.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.322+0000 7f150d4ea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:03:15.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.414+0000 7f150d4ea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:03:15.609 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.556+0000 7f150d4ea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:03:15.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2897073311' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:03:15.643 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2897073311' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-04T16:03:15.643 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:15.399+0000 7fcecb4be200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-04T16:03:15.643 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:15.495+0000 7fcecb4be200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-04T16:03:15.643 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:15.641+0000 7fcecb4be200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-04T16:03:15.722 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6800","nonce":4073253154}]},"active_addr":"172.21.15.38:6800/4073253154","active_change":"2024-03-04T15:58:15.846834+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3142726897}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":500449596}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1665273468}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":452578506}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":501976220}]}]} 2024-03-04T16:03:15.726 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-04T16:03:15.726 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T16:03:15.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:03:15.975 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:15.993 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:15.747+0000 7fcecb4be200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:03:15.993 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:15.835+0000 7fcecb4be200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:03:15.996 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.652+0000 7f150d4ea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-04T16:03:15.996 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.735+0000 7f150d4ea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-04T16:03:16.282 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:15.995+0000 7f150d4ea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:03:16.282 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:16.103+0000 7f150d4ea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:03:16.282 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:16.187+0000 7f150d4ea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:03:16.282 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:16.280+0000 7f150d4ea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:03:16.419 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:16 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:16.119+0000 7fcecb4be200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-04T16:03:16.419 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:16 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:16.234+0000 7fcecb4be200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-04T16:03:16.420 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:16 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:16.328+0000 7fcecb4be200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-04T16:03:16.608 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:16.455+0000 7f150d4ea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:03:16.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:16 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:16.427+0000 7fcecb4be200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-04T16:03:16.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:16 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:16.614+0000 7fcecb4be200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-04T16:03:17.830 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:17.407+0000 7f150d4ea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:03:17.830 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:17.535+0000 7f150d4ea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:03:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[29779]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-mon[38113]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:17.994 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:17.616+0000 7fcecb4be200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-04T16:03:17.994 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:17 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:17.760+0000 7fcecb4be200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-04T16:03:18.084 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:03:18.084 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":78,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:57:15.255572+0000","modified":"2024-03-04T16:03:16.824118+0000","last_up_change":"2024-03-04T16:02:30.058990+0000","last_in_change":"2024-03-04T16:02:12.888408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:00:32.201931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-04T16:02:41.373159+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-04T16:02:42.125847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-04T16:02:44.027394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-04T16:02:46.000786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-04T16:02:48.076921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"99ecf38d-7bc9-412d-a194-f3d9eed8e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6801","nonce":1080993770}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6802","nonce":1080993770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6804","nonce":1080993770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6803","nonce":1080993770}]},"public_addr":"172.21.15.38:6801/1080993770","cluster_addr":"172.21.15.38:6802/1080993770","heartbeat_back_addr":"172.21.15.38:6804/1080993770","heartbeat_front_addr":"172.21.15.38:6803/1080993770","state":["exists","up"]},{"osd":1,"uuid":"f8c66eb4-4966-45ce-af41-b6d01713eb51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6805","nonce":2331598100}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6806","nonce":2331598100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6808","nonce":2331598100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6807","nonce":2331598100}]},"public_addr":"172.21.15.38:6805/2331598100","cluster_addr":"172.21.15.38:6806/2331598100","heartbeat_back_addr":"172.21.15.38:6808/2331598100","heartbeat_front_addr":"172.21.15.38:6807/2331598100","state":["exists","up"]},{"osd":2,"uuid":"afb2dd7e-d27a-4536-bb80-e5724b39ffd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6809","nonce":1898914368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6810","nonce":1898914368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6812","nonce":1898914368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6811","nonce":1898914368}]},"public_addr":"172.21.15.38:6809/1898914368","cluster_addr":"172.21.15.38:6810/1898914368","heartbeat_back_addr":"172.21.15.38:6812/1898914368","heartbeat_front_addr":"172.21.15.38:6811/1898914368","state":["exists","up"]},{"osd":3,"uuid":"13f08b82-22d5-4466-9ad4-27ffa15ed7c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6813","nonce":552035408}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6814","nonce":552035408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6816","nonce":552035408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6815","nonce":552035408}]},"public_addr":"172.21.15.38:6813/552035408","cluster_addr":"172.21.15.38:6814/552035408","heartbeat_back_addr":"172.21.15.38:6816/552035408","heartbeat_front_addr":"172.21.15.38:6815/552035408","state":["exists","up"]},{"osd":4,"uuid":"a58d697b-e1de-44cc-b791-8a8d68fd01f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6800","nonce":3997489718}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6801","nonce":3997489718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6803","nonce":3997489718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6802","nonce":3997489718}]},"public_addr":"172.21.15.176:6800/3997489718","cluster_addr":"172.21.15.176:6801/3997489718","heartbeat_back_addr":"172.21.15.176:6803/3997489718","heartbeat_front_addr":"172.21.15.176:6802/3997489718","state":["exists","up"]},{"osd":5,"uuid":"7adb17cc-4843-401e-a7fd-b726c4a99a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6804","nonce":1267814386}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6805","nonce":1267814386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6807","nonce":1267814386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6806","nonce":1267814386}]},"public_addr":"172.21.15.176:6804/1267814386","cluster_addr":"172.21.15.176:6805/1267814386","heartbeat_back_addr":"172.21.15.176:6807/1267814386","heartbeat_front_addr":"172.21.15.176:6806/1267814386","state":["exists","up"]},{"osd":6,"uuid":"0e12d14b-c02d-4acf-9784-8680f8ee40f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6808","nonce":587062426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6809","nonce":587062426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6811","nonce":587062426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6810","nonce":587062426}]},"public_addr":"172.21.15.176:6808/587062426","cluster_addr":"172.21.15.176:6809/587062426","heartbeat_back_addr":"172.21.15.176:6811/587062426","heartbeat_front_addr":"172.21.15.176:6810/587062426","state":["exists","up"]},{"osd":7,"uuid":"1f348507-1649-4e25-b5f5-cb67c6cd045e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6812","nonce":1019151537}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6813","nonce":1019151537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6815","nonce":1019151537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6814","nonce":1019151537}]},"public_addr":"172.21.15.176:6812/1019151537","cluster_addr":"172.21.15.176:6813/1019151537","heartbeat_back_addr":"172.21.15.176:6815/1019151537","heartbeat_front_addr":"172.21.15.176:6814/1019151537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:23.477799+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:33.985292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:34.678790+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:02.988045+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:36.299934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:03:01.564036+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:03:16.773456+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:28.516206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/175394725":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/1249306019":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/2156780629":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/1997350354":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2476964715":"2024-03-05T15:58:15.846725+0000","172.21.15.38:0/1178105870":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/3751720132":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2774521881":"2024-03-05T15:58:15.846725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:03:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-mon[33619]: osdmap e78: 8 total, 8 up, 8 in 2024-03-04T16:03:18.109 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:17.868+0000 7f150d4ea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:03:18.109 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:17.957+0000 7f150d4ea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:03:18.419 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:18.121+0000 7fcecb4be200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-04T16:03:18.420 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:18.214+0000 7fcecb4be200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-04T16:03:18.420 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:18.418+0000 7fcecb4be200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:03:18.608 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:18.153+0000 7f150d4ea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:03:18.608 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:18.336+0000 7f150d4ea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:03:18.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:18.612+0000 7fcecb4be200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-04T16:03:18.833 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T16:03:18.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:03:18.883 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3604555971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:18.883 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:18.619+0000 7f150d4ea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:03:19.087 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:19.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3604555971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:19.191 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3604555971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:19.191 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:18 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:18.916+0000 7fcecb4be200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-04T16:03:19.191 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:19 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:19.189+0000 7fcecb4be200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:03:19.358 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:18.881+0000 7f150d4ea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-04T16:03:19.358 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:19.115+0000 7f150d4ea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:03:19.449 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:19 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:19.447+0000 7fcecb4be200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-04T16:03:20.358 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:19.910+0000 7f150d4ea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:03:20.533 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:20 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:20.297+0000 7fcecb4be200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-04T16:03:20.785 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:03:20.785 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:57:15.255572+0000","modified":"2024-03-04T16:03:20.069173+0000","last_up_change":"2024-03-04T16:02:30.058990+0000","last_in_change":"2024-03-04T16:02:12.888408+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:00:32.201931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-04T16:02:41.373159+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-04T16:02:42.125847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-04T16:02:44.027394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-04T16:02:46.000786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-04T16:02:48.076921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"99ecf38d-7bc9-412d-a194-f3d9eed8e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6801","nonce":1080993770}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6802","nonce":1080993770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6804","nonce":1080993770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6803","nonce":1080993770}]},"public_addr":"172.21.15.38:6801/1080993770","cluster_addr":"172.21.15.38:6802/1080993770","heartbeat_back_addr":"172.21.15.38:6804/1080993770","heartbeat_front_addr":"172.21.15.38:6803/1080993770","state":["exists","up"]},{"osd":1,"uuid":"f8c66eb4-4966-45ce-af41-b6d01713eb51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6805","nonce":2331598100}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6806","nonce":2331598100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6808","nonce":2331598100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6807","nonce":2331598100}]},"public_addr":"172.21.15.38:6805/2331598100","cluster_addr":"172.21.15.38:6806/2331598100","heartbeat_back_addr":"172.21.15.38:6808/2331598100","heartbeat_front_addr":"172.21.15.38:6807/2331598100","state":["exists","up"]},{"osd":2,"uuid":"afb2dd7e-d27a-4536-bb80-e5724b39ffd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6809","nonce":1898914368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6810","nonce":1898914368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6812","nonce":1898914368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6811","nonce":1898914368}]},"public_addr":"172.21.15.38:6809/1898914368","cluster_addr":"172.21.15.38:6810/1898914368","heartbeat_back_addr":"172.21.15.38:6812/1898914368","heartbeat_front_addr":"172.21.15.38:6811/1898914368","state":["exists","up"]},{"osd":3,"uuid":"13f08b82-22d5-4466-9ad4-27ffa15ed7c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6813","nonce":552035408}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6814","nonce":552035408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6816","nonce":552035408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6815","nonce":552035408}]},"public_addr":"172.21.15.38:6813/552035408","cluster_addr":"172.21.15.38:6814/552035408","heartbeat_back_addr":"172.21.15.38:6816/552035408","heartbeat_front_addr":"172.21.15.38:6815/552035408","state":["exists","up"]},{"osd":4,"uuid":"a58d697b-e1de-44cc-b791-8a8d68fd01f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6800","nonce":3997489718}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6801","nonce":3997489718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6803","nonce":3997489718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6802","nonce":3997489718}]},"public_addr":"172.21.15.176:6800/3997489718","cluster_addr":"172.21.15.176:6801/3997489718","heartbeat_back_addr":"172.21.15.176:6803/3997489718","heartbeat_front_addr":"172.21.15.176:6802/3997489718","state":["exists","up"]},{"osd":5,"uuid":"7adb17cc-4843-401e-a7fd-b726c4a99a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6804","nonce":1267814386}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6805","nonce":1267814386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6807","nonce":1267814386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6806","nonce":1267814386}]},"public_addr":"172.21.15.176:6804/1267814386","cluster_addr":"172.21.15.176:6805/1267814386","heartbeat_back_addr":"172.21.15.176:6807/1267814386","heartbeat_front_addr":"172.21.15.176:6806/1267814386","state":["exists","up"]},{"osd":6,"uuid":"0e12d14b-c02d-4acf-9784-8680f8ee40f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6808","nonce":587062426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6809","nonce":587062426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6811","nonce":587062426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6810","nonce":587062426}]},"public_addr":"172.21.15.176:6808/587062426","cluster_addr":"172.21.15.176:6809/587062426","heartbeat_back_addr":"172.21.15.176:6811/587062426","heartbeat_front_addr":"172.21.15.176:6810/587062426","state":["exists","up"]},{"osd":7,"uuid":"1f348507-1649-4e25-b5f5-cb67c6cd045e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6812","nonce":1019151537}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6813","nonce":1019151537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6815","nonce":1019151537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6814","nonce":1019151537}]},"public_addr":"172.21.15.176:6812/1019151537","cluster_addr":"172.21.15.176:6813/1019151537","heartbeat_back_addr":"172.21.15.176:6815/1019151537","heartbeat_front_addr":"172.21.15.176:6814/1019151537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:23.477799+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:33.985292+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:34.678790+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:02.988045+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:36.299934+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:03:01.564036+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:03:16.773456+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:02:28.516206+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/175394725":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/1249306019":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/2156780629":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/1997350354":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2476964715":"2024-03-05T15:58:15.846725+0000","172.21.15.38:0/1178105870":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/3751720132":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2774521881":"2024-03-05T15:58:15.846725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:03:20.795 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:20.480+0000 7f150d4ea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:03:20.795 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:20.579+0000 7f150d4ea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:03:20.795 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:20.794+0000 7f150d4ea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:03:21.070 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:20 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:20.915+0000 7fcecb4be200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-04T16:03:21.070 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:21.018+0000 7fcecb4be200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: [04/Mar/2024:16:03:20] ENGINE Bus STARTING 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: CherryPy Checker: 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: The Application mounted at '' has an empty config. 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: [04/Mar/2024:16:03:20] ENGINE Serving on http://:::9283 2024-03-04T16:03:21.077 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:03:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x[35475]: [04/Mar/2024:16:03:20] ENGINE Bus STARTED 2024-03-04T16:03:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/599823419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: Standby manager daemon x restarted 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: Standby manager daemon x started 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:03:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:21 smithi176 ceph-mon[33619]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:21.360 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:21.250+0000 7fcecb4be200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-04T16:03:21.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:21.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/599823419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:21.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: Standby manager daemon x restarted 2024-03-04T16:03:21.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: Standby manager daemon x started 2024-03-04T16:03:21.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:03:21.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:21.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:03:21.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[29779]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:21.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: osdmap e79: 8 total, 8 up, 8 in 2024-03-04T16:03:21.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/599823419' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:03:21.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: Standby manager daemon x restarted 2024-03-04T16:03:21.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: Standby manager daemon x started 2024-03-04T16:03:21.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-04T16:03:21.363 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-04T16:03:21.363 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-04T16:03:21.363 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-mon[38113]: from='mgr.? v1:172.21.15.176:0/3409015141' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-04T16:03:21.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T16:03:21.509 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T16:03:21.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T16:03:21.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T16:03:21.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T16:03:21.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T16:03:21.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-04T16:03:21.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-04T16:03:21.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:03:21] ENGINE Bus STARTING 2024-03-04T16:03:21.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: CherryPy Checker: 2024-03-04T16:03:21.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: The Application mounted at '' has an empty config. 2024-03-04T16:03:21.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:03:21.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:03:21] ENGINE Serving on http://:::9283 2024-03-04T16:03:21.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:21 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:03:21] ENGINE Bus STARTED 2024-03-04T16:03:22.051 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.170 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.170 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.170 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.171 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.171 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.171 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.171 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: Active manager daemon y restarted 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: Activating manager daemon y 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: mgrmap e19: y(active, starting, since 0.00792598s), standbys: x 2024-03-04T16:03:22.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:03:22.348 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: Manager daemon y is now available 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:22.349 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: Active manager daemon y restarted 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: Activating manager daemon y 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: mgrmap e19: y(active, starting, since 0.00792598s), standbys: x 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:03:22.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: Manager daemon y is now available 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:22.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:22.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:22.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: Active manager daemon y restarted 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: Activating manager daemon y 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: osdmap e80: 8 total, 8 up, 8 in 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: mgrmap e19: y(active, starting, since 0.00792598s), standbys: x 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-04T16:03:22.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-04T16:03:22.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: Manager daemon y is now available 2024-03-04T16:03:22.361 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:22.362 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-04T16:03:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: [04/Mar/2024:16:03:22] ENGINE Bus STARTING 2024-03-04T16:03:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: [04/Mar/2024:16:03:22] ENGINE Serving on http://172.21.15.38:8765 2024-03-04T16:03:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: [04/Mar/2024:16:03:22] ENGINE Serving on https://172.21.15.38:7150 2024-03-04T16:03:23.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: [04/Mar/2024:16:03:22] ENGINE Bus STARTED 2024-03-04T16:03:23.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: mgrmap e20: y(active, since 1.01594s), standbys: x 2024-03-04T16:03:23.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:23.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:23 smithi176 ceph-mon[33619]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:23.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:03:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: [04/Mar/2024:16:03:22] ENGINE Bus STARTING 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: [04/Mar/2024:16:03:22] ENGINE Serving on http://172.21.15.38:8765 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: [04/Mar/2024:16:03:22] ENGINE Serving on https://172.21.15.38:7150 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: [04/Mar/2024:16:03:22] ENGINE Bus STARTED 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: mgrmap e20: y(active, since 1.01594s), standbys: x 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[29779]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: [04/Mar/2024:16:03:22] ENGINE Bus STARTING 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: [04/Mar/2024:16:03:22] ENGINE Serving on http://172.21.15.38:8765 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: [04/Mar/2024:16:03:22] ENGINE Serving on https://172.21.15.38:7150 2024-03-04T16:03:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: [04/Mar/2024:16:03:22] ENGINE Bus STARTED 2024-03-04T16:03:23.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: mgrmap e20: y(active, since 1.01594s), standbys: x 2024-03-04T16:03:23.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:23.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:23 smithi038 ceph-mon[38113]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:24 smithi176 ceph-mon[33619]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:24.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:24 smithi038 ceph-mon[29779]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:24.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:24 smithi038 ceph-mon[38113]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:25 smithi038 ceph-mon[29779]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-04T16:03:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:25 smithi038 ceph-mon[29779]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:25.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:25 smithi038 ceph-mon[38113]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-04T16:03:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:25 smithi038 ceph-mon[38113]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:25 smithi176 ceph-mon[33619]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-04T16:03:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:25 smithi176 ceph-mon[33619]: osdmap e81: 8 total, 8 up, 8 in 2024-03-04T16:03:27.304 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[29779]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[29779]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[38113]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[38113]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:27.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:27 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:27 smithi176 ceph-mon[33619]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:27 smithi176 ceph-mon[33619]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-04T16:03:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:27 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:27.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:27 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.550 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.550 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.550 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:28.550 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.550 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 3760M 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:28.551 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:28 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:28.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:28.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 3760M 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:28.562 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 3760M 2024-03-04T16:03:28.563 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:28.564 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:28.564 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:28 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:29 smithi176 ceph-mon[33619]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:29.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:29 smithi038 ceph-mon[29779]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:29.670 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:29 smithi038 ceph-mon[38113]: osdmap e82: 8 total, 8 up, 8 in 2024-03-04T16:03:31.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:31 smithi176 ceph-mon[33619]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:03:31.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:31 smithi176 ceph-mon[33619]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:31.387 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:31 smithi038 ceph-mon[29779]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:03:31.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:31 smithi038 ceph-mon[29779]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:31.388 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:31 smithi038 ceph-mon[38113]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-04T16:03:31.388 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:31 smithi038 ceph-mon[38113]: osdmap e83: 8 total, 8 up, 8 in 2024-03-04T16:03:31.797 INFO:teuthology.orchestra.run.smithi038.stdout:214748364816 2024-03-04T16:03:31.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-04T16:03:32.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:03:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:03:33.187 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[29779]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-04T16:03:33.187 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:33.187 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:33.188 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[29779]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:33.188 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[38113]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-04T16:03:33.188 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:33.188 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:33.188 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-mon[38113]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:33.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:33 smithi176 ceph-mon[33619]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 19 op/s 2024-03-04T16:03:33.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:33 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:33.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:33 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:33.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:33 smithi176 ceph-mon[33619]: osdmap e84: 8 total, 8 up, 8 in 2024-03-04T16:03:33.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:03:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:03:34.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:34.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:34 smithi176 ceph-mon[33619]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-04T16:03:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:34 smithi038 ceph-mon[29779]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-04T16:03:34.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:34.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:34 smithi038 ceph-mon[38113]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-04T16:03:34.604 INFO:teuthology.orchestra.run.smithi038.stdout:137438953499 2024-03-04T16:03:34.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T16:03:36.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:36 smithi038 ceph-mon[29779]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-03-04T16:03:36.488 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:36 smithi038 ceph-mon[38113]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-03-04T16:03:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:36 smithi176 ceph-mon[33619]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 733 B/s rd, 0 op/s 2024-03-04T16:03:36.922 INFO:teuthology.orchestra.run.smithi038.stdout:180388626456 2024-03-04T16:03:36.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T16:03:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:37.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:37.733 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:37.760 INFO:teuthology.orchestra.run.smithi038.stdout:77309411366 2024-03-04T16:03:37.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T16:03:38.190 INFO:teuthology.orchestra.run.smithi038.stdout:111669149729 2024-03-04T16:03:38.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T16:03:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:38 smithi038 ceph-mon[29779]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:38.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:38 smithi038 ceph-mon[38113]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:38 smithi176 ceph-mon[33619]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:38.717 INFO:teuthology.orchestra.run.smithi038.stdout:253403070478 2024-03-04T16:03:38.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-04T16:03:39.401 INFO:teuthology.orchestra.run.smithi038.stdout:55834574892 2024-03-04T16:03:39.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T16:03:39.721 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:39.740 INFO:teuthology.orchestra.run.smithi038.stdout:34359738416 2024-03-04T16:03:39.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T16:03:40.043 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:40.320 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:41.015 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:41.016 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:41.016 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:41.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:41 smithi038 ceph-mon[29779]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:41.352 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:41 smithi038 ceph-mon[38113]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:41 smithi176 ceph-mon[33619]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:03:42.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:03:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:03:43.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:43 smithi038 ceph-mon[29779]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:43.129 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:43 smithi038 ceph-mon[38113]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:43.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:43 smithi176 ceph-mon[33619]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:43.431 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:03:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:03:44.146 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:44.147 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:45 smithi176 ceph-mon[33619]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:03:45.396 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:45 smithi038 ceph-mon[29779]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:03:45.396 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:45 smithi038 ceph-mon[38113]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:03:45.551 INFO:teuthology.orchestra.run.smithi038.stdout:77309411368 2024-03-04T16:03:45.566 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:45.584 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:03:45.996 INFO:teuthology.orchestra.run.smithi038.stdout:55834574894 2024-03-04T16:03:46.453 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4260075147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2801874499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:03:46.454 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4260075147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:03:46.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2801874499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:03:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:03:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4260075147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:03:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2801874499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:03:47.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:03:47.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:03:47.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[29779]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:47.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:03:47.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:03:47.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:47 smithi038 ceph-mon[38113]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:47 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:03:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:47 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:03:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:47 smithi176 ceph-mon[33619]: osdmap e85: 8 total, 8 up, 8 in 2024-03-04T16:03:48.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 3.10 deep-scrub starts 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 3.10 deep-scrub ok 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[29779]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:48.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 3.10 deep-scrub starts 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 3.10 deep-scrub ok 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:03:48.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:48 smithi038 ceph-mon[38113]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 3.10 deep-scrub starts 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 3.10 deep-scrub ok 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:03:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:48 smithi176 ceph-mon[33619]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:03:49.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:03:49.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:03:49.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:03:49.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:03:49.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:49 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574894 for osd.1 2024-03-04T16:03:49.520 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:03:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:03:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:03:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:03:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:03:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:03:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:49 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:49.955 INFO:teuthology.orchestra.run.smithi038.stdout:180388626459 2024-03-04T16:03:50.277 INFO:teuthology.orchestra.run.smithi038.stdout:111669149732 2024-03-04T16:03:50.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:03:50.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:03:50.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2134575217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[29779]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2134575217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:03:50.292 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:50 smithi038 ceph-mon[38113]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:03:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:03:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:03:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:03:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:03:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2134575217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:03:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:50 smithi176 ceph-mon[33619]: osdmap e86: 8 total, 8 up, 8 in 2024-03-04T16:03:51.089 INFO:teuthology.orchestra.run.smithi038.stdout:214748364821 2024-03-04T16:03:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3018217307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:03:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 3761M 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/878660740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:03:51.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3018217307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 3761M 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:51.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/878660740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3018217307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:03:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 3761M 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi176:/etc/ceph/ceph.conf 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: Updating smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:51.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/878660740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:03:52.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411368 for osd.2 2024-03-04T16:03:52.341 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:03:52.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: Deploying daemon alertmanager.a on smithi038 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:03:52.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.448 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: Deploying daemon alertmanager.a on smithi038 2024-03-04T16:03:52.449 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:52.449 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:52.449 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:52 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:52.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: Updating smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:03:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: Deploying daemon alertmanager.a on smithi038 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:52.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:52 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:53.370 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364816 got 214748364821 for osd.6 2024-03-04T16:03:53.370 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:53.450 INFO:teuthology.orchestra.run.smithi038.stdout:34359738419 2024-03-04T16:03:53.451 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:03:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:03:53.451 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:03:53.451 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:03:53.451 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:03:53.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[29779]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:03:53.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:03:53.454 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:53 smithi038 ceph-mon[38113]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:03:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:53 smithi176 ceph-mon[33619]: osdmap e87: 8 total, 8 up, 8 in 2024-03-04T16:03:53.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149732 for osd.3 2024-03-04T16:03:53.824 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:54.299 INFO:teuthology.orchestra.run.smithi038.stdout:137438953504 2024-03-04T16:03:54.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 4.17 deep-scrub starts 2024-03-04T16:03:54.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 4.17 deep-scrub ok 2024-03-04T16:03:54.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:03:54.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 6.4 deep-scrub starts 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: 6.4 deep-scrub ok 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3235048815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 4.17 deep-scrub starts 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 4.17 deep-scrub ok 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 6.4 deep-scrub starts 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: 6.4 deep-scrub ok 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:03:54.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3235048815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:03:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 4.17 deep-scrub starts 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 4.17 deep-scrub ok 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 6.4 deep-scrub starts 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: 6.4 deep-scrub ok 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:03:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3235048815' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2279793659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:03:55.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:03:55.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[29779]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:55.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:03:55.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2279793659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:03:55.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:03:55.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:03:55.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:55 smithi038 ceph-mon[38113]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:55.328 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-03-04T16:03:55.329 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2279793659' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:03:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:55 smithi176 ceph-mon[33619]: osdmap e88: 8 total, 8 up, 8 in 2024-03-04T16:03:56.288 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626456 got 180388626459 for osd.5 2024-03-04T16:03:56.288 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[29779]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:56.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:03:56.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:03:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[38113]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:03:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:56 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:03:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:56 smithi176 ceph-mon[33619]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:03:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:56 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:03:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:56 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:03:57.452 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953499 got 137438953504 for osd.4 2024-03-04T16:03:57.452 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:03:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:03:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:57 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:03:57.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:57 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:03:57.970 INFO:teuthology.orchestra.run.smithi038.stdout:253403070482 2024-03-04T16:03:58.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:03:58.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:03:58.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:03:58.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:03:58.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:03:58.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1483399202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:03:58.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:03:58.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:03:58.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:03:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1483399202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:03:58.526 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:03:58.526 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: osdmap e89: 8 total, 8 up, 8 in 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:03:58.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1483399202' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:03:59.461 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070478 got 253403070482 for osd.7 2024-03-04T16:03:59.462 DEBUG:teuthology.parallel:result is None 2024-03-04T16:03:59.462 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T16:03:59.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:03:59.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:03:59.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:03:59 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:03:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:03:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:03:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:03:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:03:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:03:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:03:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:03:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:03:59 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:04:00.277 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:00.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:04:00.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:04:00.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:04:00.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:04:00.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[38113]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-04T16:04:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:00 smithi038 ceph-mon[29779]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:04:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-04T16:04:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:00 smithi176 ceph-mon[33619]: osdmap e90: 8 total, 8 up, 8 in 2024-03-04T16:04:01.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:04:01.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:04:01.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:04:01.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:04:01.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:04:01.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:04:01.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:04:01.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:01 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:04:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:01 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:04:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:01 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:04:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:01 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:04:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:01 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:04:02.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[29779]: 4.1a deep-scrub starts 2024-03-04T16:04:02.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[29779]: 4.1a deep-scrub ok 2024-03-04T16:04:02.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:04:02.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:04:02.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[29779]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-04T16:04:02.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[38113]: 4.1a deep-scrub starts 2024-03-04T16:04:02.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[38113]: 4.1a deep-scrub ok 2024-03-04T16:04:02.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:04:02.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:04:02.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:02 smithi038 ceph-mon[38113]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-04T16:04:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-mon[33619]: 4.1a deep-scrub starts 2024-03-04T16:04:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-mon[33619]: 4.1a deep-scrub ok 2024-03-04T16:04:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:04:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:04:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-mon[33619]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-04T16:04:02.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:04:03.354 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:03 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:04:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:03 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:04:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:03.947 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:04:04.243 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 podman[73256]: 2024-03-04 16:04:04.091264674 +0000 UTC m=+0.213720609 container create 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[29779]: from='client.34640 v1:172.21.15.38:0/2807552974' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:04.511 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:04:04.512 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:04:04.512 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:04.512 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:04:04.512 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:04:04.512 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-mon[38113]: from='client.34640 v1:172.21.15.38:0/2807552974' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:04.512 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 podman[73256]: 2024-03-04 16:04:04.411246853 +0000 UTC m=+0.533702794 container init 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:04.512 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 podman[73256]: 2024-03-04 16:04:04.414275649 +0000 UTC m=+0.536731585 container start 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:04.512 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.426Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-04T16:04:04.512 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.427Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-04T16:04:04.512 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.427Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.38 port=9094 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.428Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.452Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.452Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.455Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:04.455Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-04T16:04:04.513 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 bash[73256]: 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c 2024-03-04T16:04:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:04:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:04:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:04:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:04:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:04 smithi176 ceph-mon[33619]: from='client.34640 v1:172.21.15.38:0/2807552974' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:04.797 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:04 smithi038 systemd[1]: Started Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE Bus STOPPING 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE Bus STOPPED 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE Bus STARTING 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE Serving on http://:::9283 2024-03-04T16:04:04.798 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:04 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:04] ENGINE Bus STARTED 2024-03-04T16:04:05.223 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-04T16:04:03.290310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1007,"num_read_kb":853,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240928,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720288,"statfs":{"total":767960285184,"available":767713574912,"internally_reserved":0,"allocated":8318976,"data_stored":4614954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001714"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171404+0000","last_change":"2024-03-04T16:03:57.477127+0000","last_active":"2024-03-04T16:04:00.171404+0000","last_peered":"2024-03-04T16:04:00.171404+0000","last_clean":"2024-03-04T16:04:00.171404+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:04:00.171404+0000","last_undegraded":"2024-03-04T16:04:00.171404+0000","last_fullsized":"2024-03-04T16:04:00.171404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.477014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.477014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.118199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170834+0000","last_change":"2024-03-04T16:02:47.767863+0000","last_active":"2024-03-04T16:04:00.170834+0000","last_peered":"2024-03-04T16:04:00.170834+0000","last_clean":"2024-03-04T16:04:00.170834+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:04:00.170834+0000","last_undegraded":"2024-03-04T16:04:00.170834+0000","last_fullsized":"2024-03-04T16:04:00.170834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.395983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171558+0000","last_change":"2024-03-04T16:03:45.773330+0000","last_active":"2024-03-04T16:04:00.171558+0000","last_peered":"2024-03-04T16:04:00.171558+0000","last_clean":"2024-03-04T16:04:00.171558+0000","last_became_active":"2024-03-04T16:02:43.714846+0000","last_became_peered":"2024-03-04T16:02:43.714846+0000","last_unstale":"2024-03-04T16:04:00.171558+0000","last_undegraded":"2024-03-04T16:04:00.171558+0000","last_fullsized":"2024-03-04T16:04:00.171558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:45.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.773213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.488104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170701+0000","last_change":"2024-03-04T16:02:45.748421+0000","last_active":"2024-03-04T16:04:00.170701+0000","last_peered":"2024-03-04T16:04:00.170701+0000","last_clean":"2024-03-04T16:04:00.170701+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:04:00.170701+0000","last_undegraded":"2024-03-04T16:04:00.170701+0000","last_fullsized":"2024-03-04T16:04:00.170701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.620264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096208+0000","last_change":"2024-03-04T16:02:49.792565+0000","last_active":"2024-03-04T16:04:00.096208+0000","last_peered":"2024-03-04T16:04:00.096208+0000","last_clean":"2024-03-04T16:04:00.096208+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:04:00.096208+0000","last_undegraded":"2024-03-04T16:04:00.096208+0000","last_fullsized":"2024-03-04T16:04:00.096208+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172064+0000","last_change":"2024-03-04T16:02:47.766441+0000","last_active":"2024-03-04T16:04:00.172064+0000","last_peered":"2024-03-04T16:04:00.172064+0000","last_clean":"2024-03-04T16:04:00.172064+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:04:00.172064+0000","last_undegraded":"2024-03-04T16:04:00.172064+0000","last_fullsized":"2024-03-04T16:04:00.172064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096811+0000","last_change":"2024-03-04T16:02:43.712774+0000","last_active":"2024-03-04T16:04:00.096811+0000","last_peered":"2024-03-04T16:04:00.096811+0000","last_clean":"2024-03-04T16:04:00.096811+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:04:00.096811+0000","last_undegraded":"2024-03-04T16:04:00.096811+0000","last_fullsized":"2024-03-04T16:04:00.096811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.121882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096197+0000","last_change":"2024-03-04T16:03:50.379226+0000","last_active":"2024-03-04T16:04:00.096197+0000","last_peered":"2024-03-04T16:04:00.096197+0000","last_clean":"2024-03-04T16:04:00.096197+0000","last_became_active":"2024-03-04T16:02:45.749383+0000","last_became_peered":"2024-03-04T16:02:45.749383+0000","last_unstale":"2024-03-04T16:04:00.096197+0000","last_undegraded":"2024-03-04T16:04:00.096197+0000","last_fullsized":"2024-03-04T16:04:00.096197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:50.379077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.379077+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.153393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170489+0000","last_change":"2024-03-04T16:03:56.851189+0000","last_active":"2024-03-04T16:04:00.170489+0000","last_peered":"2024-03-04T16:04:00.170489+0000","last_clean":"2024-03-04T16:04:00.170489+0000","last_became_active":"2024-03-04T16:02:49.791142+0000","last_became_peered":"2024-03-04T16:02:49.791142+0000","last_unstale":"2024-03-04T16:04:00.170489+0000","last_undegraded":"2024-03-04T16:04:00.170489+0000","last_fullsized":"2024-03-04T16:04:00.170489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.851094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.851094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.117129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171309+0000","last_change":"2024-03-04T16:02:47.775367+0000","last_active":"2024-03-04T16:04:00.171309+0000","last_peered":"2024-03-04T16:04:00.171309+0000","last_clean":"2024-03-04T16:04:00.171309+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:04:00.171309+0000","last_undegraded":"2024-03-04T16:04:00.171309+0000","last_fullsized":"2024-03-04T16:04:00.171309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.508809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172396+0000","last_change":"2024-03-04T16:03:51.290213+0000","last_active":"2024-03-04T16:04:00.172396+0000","last_peered":"2024-03-04T16:04:00.172396+0000","last_clean":"2024-03-04T16:04:00.172396+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:04:00.172396+0000","last_undegraded":"2024-03-04T16:04:00.172396+0000","last_fullsized":"2024-03-04T16:04:00.172396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.290117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.290117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.885059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.434104+0000","last_change":"2024-03-04T16:04:00.434148+0000","last_active":"2024-03-04T16:04:00.434104+0000","last_peered":"2024-03-04T16:04:00.434104+0000","last_clean":"2024-03-04T16:04:00.434104+0000","last_became_active":"2024-03-04T16:02:45.745532+0000","last_became_peered":"2024-03-04T16:02:45.745532+0000","last_unstale":"2024-03-04T16:04:00.434104+0000","last_undegraded":"2024-03-04T16:04:00.434104+0000","last_fullsized":"2024-03-04T16:04:00.434104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.434035+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.035919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170770+0000","last_change":"2024-03-04T16:02:49.791512+0000","last_active":"2024-03-04T16:04:00.170770+0000","last_peered":"2024-03-04T16:04:00.170770+0000","last_clean":"2024-03-04T16:04:00.170770+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:04:00.170770+0000","last_undegraded":"2024-03-04T16:04:00.170770+0000","last_fullsized":"2024-03-04T16:04:00.170770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171213+0000","last_change":"2024-03-04T16:02:47.768573+0000","last_active":"2024-03-04T16:04:00.171213+0000","last_peered":"2024-03-04T16:04:00.171213+0000","last_clean":"2024-03-04T16:04:00.171213+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:04:00.171213+0000","last_undegraded":"2024-03-04T16:04:00.171213+0000","last_fullsized":"2024-03-04T16:04:00.171213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.292580+0000","last_change":"2024-03-04T16:04:00.292627+0000","last_active":"2024-03-04T16:04:00.292580+0000","last_peered":"2024-03-04T16:04:00.292580+0000","last_clean":"2024-03-04T16:04:00.292580+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:04:00.292580+0000","last_undegraded":"2024-03-04T16:04:00.292580+0000","last_fullsized":"2024-03-04T16:04:00.292580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:00.292499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.292499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.161517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171851+0000","last_change":"2024-03-04T16:02:45.748750+0000","last_active":"2024-03-04T16:04:00.171851+0000","last_peered":"2024-03-04T16:04:00.171851+0000","last_clean":"2024-03-04T16:04:00.171851+0000","last_became_active":"2024-03-04T16:02:45.747787+0000","last_became_peered":"2024-03-04T16:02:45.747787+0000","last_unstale":"2024-03-04T16:04:00.171851+0000","last_undegraded":"2024-03-04T16:04:00.171851+0000","last_fullsized":"2024-03-04T16:04:00.171851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:14.663457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172933+0000","last_change":"2024-03-04T16:02:49.785706+0000","last_active":"2024-03-04T16:04:00.172933+0000","last_peered":"2024-03-04T16:04:00.172933+0000","last_clean":"2024-03-04T16:04:00.172933+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:04:00.172933+0000","last_undegraded":"2024-03-04T16:04:00.172933+0000","last_fullsized":"2024-03-04T16:04:00.172933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.708627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171854+0000","last_change":"2024-03-04T16:02:47.768091+0000","last_active":"2024-03-04T16:04:00.171854+0000","last_peered":"2024-03-04T16:04:00.171854+0000","last_clean":"2024-03-04T16:04:00.171854+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:04:00.171854+0000","last_undegraded":"2024-03-04T16:04:00.171854+0000","last_fullsized":"2024-03-04T16:04:00.171854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171158+0000","last_change":"2024-03-04T16:02:43.714700+0000","last_active":"2024-03-04T16:04:00.171158+0000","last_peered":"2024-03-04T16:04:00.171158+0000","last_clean":"2024-03-04T16:04:00.171158+0000","last_became_active":"2024-03-04T16:02:43.714419+0000","last_became_peered":"2024-03-04T16:02:43.714419+0000","last_unstale":"2024-03-04T16:04:00.171158+0000","last_undegraded":"2024-03-04T16:04:00.171158+0000","last_fullsized":"2024-03-04T16:04:00.171158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.557877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095293+0000","last_change":"2024-03-04T16:03:45.980725+0000","last_active":"2024-03-04T16:04:00.095293+0000","last_peered":"2024-03-04T16:04:00.095293+0000","last_clean":"2024-03-04T16:04:00.095293+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:04:00.095293+0000","last_undegraded":"2024-03-04T16:04:00.095293+0000","last_fullsized":"2024-03-04T16:04:00.095293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:45.980606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.980606+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:01.702710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096882+0000","last_change":"2024-03-04T16:02:49.794035+0000","last_active":"2024-03-04T16:04:00.096882+0000","last_peered":"2024-03-04T16:04:00.096882+0000","last_clean":"2024-03-04T16:04:00.096882+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:04:00.096882+0000","last_undegraded":"2024-03-04T16:04:00.096882+0000","last_fullsized":"2024-03-04T16:04:00.096882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.379614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171229+0000","last_change":"2024-03-04T16:02:47.769883+0000","last_active":"2024-03-04T16:04:00.171229+0000","last_peered":"2024-03-04T16:04:00.171229+0000","last_clean":"2024-03-04T16:04:00.171229+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:04:00.171229+0000","last_undegraded":"2024-03-04T16:04:00.171229+0000","last_fullsized":"2024-03-04T16:04:00.171229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.833777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171760+0000","last_change":"2024-03-04T16:03:46.423866+0000","last_active":"2024-03-04T16:04:00.171760+0000","last_peered":"2024-03-04T16:04:00.171760+0000","last_clean":"2024-03-04T16:04:00.171760+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:04:00.171760+0000","last_undegraded":"2024-03-04T16:04:00.171760+0000","last_fullsized":"2024-03-04T16:04:00.171760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.423767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.868075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096446+0000","last_change":"2024-03-04T16:03:55.411658+0000","last_active":"2024-03-04T16:04:00.096446+0000","last_peered":"2024-03-04T16:04:00.096446+0000","last_clean":"2024-03-04T16:04:00.096446+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:04:00.096446+0000","last_undegraded":"2024-03-04T16:04:00.096446+0000","last_fullsized":"2024-03-04T16:04:00.096446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:55.411554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.411554+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.742258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171968+0000","last_change":"2024-03-04T16:02:49.792726+0000","last_active":"2024-03-04T16:04:00.171968+0000","last_peered":"2024-03-04T16:04:00.171968+0000","last_clean":"2024-03-04T16:04:00.171968+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:04:00.171968+0000","last_undegraded":"2024-03-04T16:04:00.171968+0000","last_fullsized":"2024-03-04T16:04:00.171968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.657727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087676+0000","last_change":"2024-03-04T16:02:47.772339+0000","last_active":"2024-03-04T16:04:00.087676+0000","last_peered":"2024-03-04T16:04:00.087676+0000","last_clean":"2024-03-04T16:04:00.087676+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:04:00.087676+0000","last_undegraded":"2024-03-04T16:04:00.087676+0000","last_fullsized":"2024-03-04T16:04:00.087676+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.633218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:01.014974+0000","last_change":"2024-03-04T16:04:01.015009+0000","last_active":"2024-03-04T16:04:01.014974+0000","last_peered":"2024-03-04T16:04:01.014974+0000","last_clean":"2024-03-04T16:04:01.014974+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:04:01.014974+0000","last_undegraded":"2024-03-04T16:04:01.014974+0000","last_fullsized":"2024-03-04T16:04:01.014974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:01.014871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:01.014871+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:13.772804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171781+0000","last_change":"2024-03-04T16:02:45.745063+0000","last_active":"2024-03-04T16:04:00.171781+0000","last_peered":"2024-03-04T16:04:00.171781+0000","last_clean":"2024-03-04T16:04:00.171781+0000","last_became_active":"2024-03-04T16:02:45.744705+0000","last_became_peered":"2024-03-04T16:02:45.744705+0000","last_unstale":"2024-03-04T16:04:00.171781+0000","last_undegraded":"2024-03-04T16:04:00.171781+0000","last_fullsized":"2024-03-04T16:04:00.171781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.665716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171960+0000","last_change":"2024-03-04T16:03:53.955279+0000","last_active":"2024-03-04T16:04:00.171960+0000","last_peered":"2024-03-04T16:04:00.171960+0000","last_clean":"2024-03-04T16:04:00.171960+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:04:00.171960+0000","last_undegraded":"2024-03-04T16:04:00.171960+0000","last_fullsized":"2024-03-04T16:04:00.171960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:53.955186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:53.955186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.764558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171905+0000","last_change":"2024-03-04T16:03:57.874266+0000","last_active":"2024-03-04T16:04:00.171905+0000","last_peered":"2024-03-04T16:04:00.171905+0000","last_clean":"2024-03-04T16:04:00.171905+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:04:00.171905+0000","last_undegraded":"2024-03-04T16:04:00.171905+0000","last_fullsized":"2024-03-04T16:04:00.171905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.874150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.874150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.705755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095851+0000","last_change":"2024-03-04T16:03:47.354437+0000","last_active":"2024-03-04T16:04:00.095851+0000","last_peered":"2024-03-04T16:04:00.095851+0000","last_clean":"2024-03-04T16:04:00.095851+0000","last_became_active":"2024-03-04T16:02:43.718570+0000","last_became_peered":"2024-03-04T16:02:43.718570+0000","last_unstale":"2024-03-04T16:04:00.095851+0000","last_undegraded":"2024-03-04T16:04:00.095851+0000","last_fullsized":"2024-03-04T16:04:00.095851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.354324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.354324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.540186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087619+0000","last_change":"2024-03-04T16:03:51.720621+0000","last_active":"2024-03-04T16:04:00.087619+0000","last_peered":"2024-03-04T16:04:00.087619+0000","last_clean":"2024-03-04T16:04:00.087619+0000","last_became_active":"2024-03-04T16:02:45.746864+0000","last_became_peered":"2024-03-04T16:02:45.746864+0000","last_unstale":"2024-03-04T16:04:00.087619+0000","last_undegraded":"2024-03-04T16:04:00.087619+0000","last_fullsized":"2024-03-04T16:04:00.087619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:51.720518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.720518+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.935701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171107+0000","last_change":"2024-03-04T16:03:57.893125+0000","last_active":"2024-03-04T16:04:00.171107+0000","last_peered":"2024-03-04T16:04:00.171107+0000","last_clean":"2024-03-04T16:04:00.171107+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:04:00.171107+0000","last_undegraded":"2024-03-04T16:04:00.171107+0000","last_fullsized":"2024-03-04T16:04:00.171107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.893007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.893007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.709209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.086951+0000","last_change":"2024-03-04T16:02:47.771083+0000","last_active":"2024-03-04T16:04:00.086951+0000","last_peered":"2024-03-04T16:04:00.086951+0000","last_clean":"2024-03-04T16:04:00.086951+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:04:00.086951+0000","last_undegraded":"2024-03-04T16:04:00.086951+0000","last_fullsized":"2024-03-04T16:04:00.086951+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171475+0000","last_change":"2024-03-04T16:02:43.717093+0000","last_active":"2024-03-04T16:04:00.171475+0000","last_peered":"2024-03-04T16:04:00.171475+0000","last_clean":"2024-03-04T16:04:00.171475+0000","last_became_active":"2024-03-04T16:02:43.716542+0000","last_became_peered":"2024-03-04T16:02:43.716542+0000","last_unstale":"2024-03-04T16:04:00.171475+0000","last_undegraded":"2024-03-04T16:04:00.171475+0000","last_fullsized":"2024-03-04T16:04:00.171475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.808612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095860+0000","last_change":"2024-03-04T16:02:45.754181+0000","last_active":"2024-03-04T16:04:00.095860+0000","last_peered":"2024-03-04T16:04:00.095860+0000","last_clean":"2024-03-04T16:04:00.095860+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:04:00.095860+0000","last_undegraded":"2024-03-04T16:04:00.095860+0000","last_fullsized":"2024-03-04T16:04:00.095860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.606250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096015+0000","last_change":"2024-03-04T16:02:49.791755+0000","last_active":"2024-03-04T16:04:00.096015+0000","last_peered":"2024-03-04T16:04:00.096015+0000","last_clean":"2024-03-04T16:04:00.096015+0000","last_became_active":"2024-03-04T16:02:49.791216+0000","last_became_peered":"2024-03-04T16:02:49.791216+0000","last_unstale":"2024-03-04T16:04:00.096015+0000","last_undegraded":"2024-03-04T16:04:00.096015+0000","last_fullsized":"2024-03-04T16:04:00.096015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.261130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171216+0000","last_change":"2024-03-04T16:02:47.773727+0000","last_active":"2024-03-04T16:04:00.171216+0000","last_peered":"2024-03-04T16:04:00.171216+0000","last_clean":"2024-03-04T16:04:00.171216+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:04:00.171216+0000","last_undegraded":"2024-03-04T16:04:00.171216+0000","last_fullsized":"2024-03-04T16:04:00.171216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.258206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171414+0000","last_change":"2024-03-04T16:03:52.330168+0000","last_active":"2024-03-04T16:04:00.171414+0000","last_peered":"2024-03-04T16:04:00.171414+0000","last_clean":"2024-03-04T16:04:00.171414+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:04:00.171414+0000","last_undegraded":"2024-03-04T16:04:00.171414+0000","last_fullsized":"2024-03-04T16:04:00.171414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.330026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.330026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.392092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096129+0000","last_change":"2024-03-04T16:02:45.753666+0000","last_active":"2024-03-04T16:04:00.096129+0000","last_peered":"2024-03-04T16:04:00.096129+0000","last_clean":"2024-03-04T16:04:00.096129+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:04:00.096129+0000","last_undegraded":"2024-03-04T16:04:00.096129+0000","last_fullsized":"2024-03-04T16:04:00.096129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170320+0000","last_change":"2024-03-04T16:02:49.794500+0000","last_active":"2024-03-04T16:04:00.170320+0000","last_peered":"2024-03-04T16:04:00.170320+0000","last_clean":"2024-03-04T16:04:00.170320+0000","last_became_active":"2024-03-04T16:02:49.794197+0000","last_became_peered":"2024-03-04T16:02:49.794197+0000","last_unstale":"2024-03-04T16:04:00.170320+0000","last_undegraded":"2024-03-04T16:04:00.170320+0000","last_fullsized":"2024-03-04T16:04:00.170320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096301+0000","last_change":"2024-03-04T16:02:47.777366+0000","last_active":"2024-03-04T16:04:00.096301+0000","last_peered":"2024-03-04T16:04:00.096301+0000","last_clean":"2024-03-04T16:04:00.096301+0000","last_became_active":"2024-03-04T16:02:47.777008+0000","last_became_peered":"2024-03-04T16:02:47.777008+0000","last_unstale":"2024-03-04T16:04:00.096301+0000","last_undegraded":"2024-03-04T16:04:00.096301+0000","last_fullsized":"2024-03-04T16:04:00.096301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.335199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172131+0000","last_change":"2024-03-04T16:02:43.711547+0000","last_active":"2024-03-04T16:04:00.172131+0000","last_peered":"2024-03-04T16:04:00.172131+0000","last_clean":"2024-03-04T16:04:00.172131+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:04:00.172131+0000","last_undegraded":"2024-03-04T16:04:00.172131+0000","last_fullsized":"2024-03-04T16:04:00.172131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.527444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171111+0000","last_change":"2024-03-04T16:02:45.745623+0000","last_active":"2024-03-04T16:04:00.171111+0000","last_peered":"2024-03-04T16:04:00.171111+0000","last_clean":"2024-03-04T16:04:00.171111+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:04:00.171111+0000","last_undegraded":"2024-03-04T16:04:00.171111+0000","last_fullsized":"2024-03-04T16:04:00.171111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095861+0000","last_change":"2024-03-04T16:02:49.792155+0000","last_active":"2024-03-04T16:04:00.095861+0000","last_peered":"2024-03-04T16:04:00.095861+0000","last_clean":"2024-03-04T16:04:00.095861+0000","last_became_active":"2024-03-04T16:02:49.791815+0000","last_became_peered":"2024-03-04T16:02:49.791815+0000","last_unstale":"2024-03-04T16:04:00.095861+0000","last_undegraded":"2024-03-04T16:04:00.095861+0000","last_fullsized":"2024-03-04T16:04:00.095861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.727768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170892+0000","last_change":"2024-03-04T16:03:51.939795+0000","last_active":"2024-03-04T16:04:00.170892+0000","last_peered":"2024-03-04T16:04:00.170892+0000","last_clean":"2024-03-04T16:04:00.170892+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:04:00.170892+0000","last_undegraded":"2024-03-04T16:04:00.170892+0000","last_fullsized":"2024-03-04T16:04:00.170892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.939712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.939712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.441874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171382+0000","last_change":"2024-03-04T16:03:44.419164+0000","last_active":"2024-03-04T16:04:00.171382+0000","last_peered":"2024-03-04T16:04:00.171382+0000","last_clean":"2024-03-04T16:04:00.171382+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:04:00.171382+0000","last_undegraded":"2024-03-04T16:04:00.171382+0000","last_fullsized":"2024-03-04T16:04:00.171382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:44.419065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:44.419065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.867222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171695+0000","last_change":"2024-03-04T16:03:50.400887+0000","last_active":"2024-03-04T16:04:00.171695+0000","last_peered":"2024-03-04T16:04:00.171695+0000","last_clean":"2024-03-04T16:04:00.171695+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:04:00.171695+0000","last_undegraded":"2024-03-04T16:04:00.171695+0000","last_fullsized":"2024-03-04T16:04:00.171695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:50.400784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.400784+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.749407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171573+0000","last_change":"2024-03-04T16:02:49.797503+0000","last_active":"2024-03-04T16:04:00.171573+0000","last_peered":"2024-03-04T16:04:00.171573+0000","last_clean":"2024-03-04T16:04:00.171573+0000","last_became_active":"2024-03-04T16:02:49.797174+0000","last_became_peered":"2024-03-04T16:02:49.797174+0000","last_unstale":"2024-03-04T16:04:00.171573+0000","last_undegraded":"2024-03-04T16:04:00.171573+0000","last_fullsized":"2024-03-04T16:04:00.171573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:17.229451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172495+0000","last_change":"2024-03-04T16:02:47.769130+0000","last_active":"2024-03-04T16:04:00.172495+0000","last_peered":"2024-03-04T16:04:00.172495+0000","last_clean":"2024-03-04T16:04:00.172495+0000","last_became_active":"2024-03-04T16:02:47.768874+0000","last_became_peered":"2024-03-04T16:02:47.768874+0000","last_unstale":"2024-03-04T16:04:00.172495+0000","last_undegraded":"2024-03-04T16:04:00.172495+0000","last_fullsized":"2024-03-04T16:04:00.172495+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.540758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171510+0000","last_change":"2024-03-04T16:02:43.709781+0000","last_active":"2024-03-04T16:04:00.171510+0000","last_peered":"2024-03-04T16:04:00.171510+0000","last_clean":"2024-03-04T16:04:00.171510+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:04:00.171510+0000","last_undegraded":"2024-03-04T16:04:00.171510+0000","last_fullsized":"2024-03-04T16:04:00.171510+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096389+0000","last_change":"2024-03-04T16:02:45.749382+0000","last_active":"2024-03-04T16:04:00.096389+0000","last_peered":"2024-03-04T16:04:00.096389+0000","last_clean":"2024-03-04T16:04:00.096389+0000","last_became_active":"2024-03-04T16:02:45.748824+0000","last_became_peered":"2024-03-04T16:02:45.748824+0000","last_unstale":"2024-03-04T16:04:00.096389+0000","last_undegraded":"2024-03-04T16:04:00.096389+0000","last_fullsized":"2024-03-04T16:04:00.096389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.173149+0000","last_change":"2024-03-04T16:02:49.787819+0000","last_active":"2024-03-04T16:04:00.173149+0000","last_peered":"2024-03-04T16:04:00.173149+0000","last_clean":"2024-03-04T16:04:00.173149+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:04:00.173149+0000","last_undegraded":"2024-03-04T16:04:00.173149+0000","last_fullsized":"2024-03-04T16:04:00.173149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.531354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096648+0000","last_change":"2024-03-04T16:02:47.774221+0000","last_active":"2024-03-04T16:04:00.096648+0000","last_peered":"2024-03-04T16:04:00.096648+0000","last_clean":"2024-03-04T16:04:00.096648+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:04:00.096648+0000","last_undegraded":"2024-03-04T16:04:00.096648+0000","last_fullsized":"2024-03-04T16:04:00.096648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.715311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170631+0000","last_change":"2024-03-04T16:03:49.810121+0000","last_active":"2024-03-04T16:04:00.170631+0000","last_peered":"2024-03-04T16:04:00.170631+0000","last_clean":"2024-03-04T16:04:00.170631+0000","last_became_active":"2024-03-04T16:02:43.713331+0000","last_became_peered":"2024-03-04T16:02:43.713331+0000","last_unstale":"2024-03-04T16:04:00.170631+0000","last_undegraded":"2024-03-04T16:04:00.170631+0000","last_fullsized":"2024-03-04T16:04:00.170631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:03:49.810002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:49.810002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.948952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172670+0000","last_change":"2024-03-04T16:03:55.306308+0000","last_active":"2024-03-04T16:04:00.172670+0000","last_peered":"2024-03-04T16:04:00.172670+0000","last_clean":"2024-03-04T16:04:00.172670+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:04:00.172670+0000","last_undegraded":"2024-03-04T16:04:00.172670+0000","last_fullsized":"2024-03-04T16:04:00.172670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:03:55.306180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.306180+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.830154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095503+0000","last_change":"2024-03-04T16:02:49.790101+0000","last_active":"2024-03-04T16:04:00.095503+0000","last_peered":"2024-03-04T16:04:00.095503+0000","last_clean":"2024-03-04T16:04:00.095503+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:04:00.095503+0000","last_undegraded":"2024-03-04T16:04:00.095503+0000","last_fullsized":"2024-03-04T16:04:00.095503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.704953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096828+0000","last_change":"2024-03-04T16:02:47.775318+0000","last_active":"2024-03-04T16:04:00.096828+0000","last_peered":"2024-03-04T16:04:00.096828+0000","last_clean":"2024-03-04T16:04:00.096828+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:04:00.096828+0000","last_undegraded":"2024-03-04T16:04:00.096828+0000","last_fullsized":"2024-03-04T16:04:00.096828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171544+0000","last_change":"2024-03-04T16:02:43.714055+0000","last_active":"2024-03-04T16:04:00.171544+0000","last_peered":"2024-03-04T16:04:00.171544+0000","last_clean":"2024-03-04T16:04:00.171544+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:04:00.171544+0000","last_undegraded":"2024-03-04T16:04:00.171544+0000","last_fullsized":"2024-03-04T16:04:00.171544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.115403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171807+0000","last_change":"2024-03-04T16:02:45.745930+0000","last_active":"2024-03-04T16:04:00.171807+0000","last_peered":"2024-03-04T16:04:00.171807+0000","last_clean":"2024-03-04T16:04:00.171807+0000","last_became_active":"2024-03-04T16:02:45.745571+0000","last_became_peered":"2024-03-04T16:02:45.745571+0000","last_unstale":"2024-03-04T16:04:00.171807+0000","last_undegraded":"2024-03-04T16:04:00.171807+0000","last_fullsized":"2024-03-04T16:04:00.171807+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.289714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172040+0000","last_change":"2024-03-04T16:02:49.787602+0000","last_active":"2024-03-04T16:04:00.172040+0000","last_peered":"2024-03-04T16:04:00.172040+0000","last_clean":"2024-03-04T16:04:00.172040+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:04:00.172040+0000","last_undegraded":"2024-03-04T16:04:00.172040+0000","last_fullsized":"2024-03-04T16:04:00.172040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.184909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171501+0000","last_change":"2024-03-04T16:02:47.770162+0000","last_active":"2024-03-04T16:04:00.171501+0000","last_peered":"2024-03-04T16:04:00.171501+0000","last_clean":"2024-03-04T16:04:00.171501+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:04:00.171501+0000","last_undegraded":"2024-03-04T16:04:00.171501+0000","last_fullsized":"2024-03-04T16:04:00.171501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.086907+0000","last_change":"2024-03-04T16:03:46.718298+0000","last_active":"2024-03-04T16:04:00.086907+0000","last_peered":"2024-03-04T16:04:00.086907+0000","last_clean":"2024-03-04T16:04:00.086907+0000","last_became_active":"2024-03-04T16:02:43.718241+0000","last_became_peered":"2024-03-04T16:02:43.718241+0000","last_unstale":"2024-03-04T16:04:00.086907+0000","last_undegraded":"2024-03-04T16:04:00.086907+0000","last_fullsized":"2024-03-04T16:04:00.086907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.718188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.606455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096922+0000","last_change":"2024-03-04T16:03:52.421377+0000","last_active":"2024-03-04T16:04:00.096922+0000","last_peered":"2024-03-04T16:04:00.096922+0000","last_clean":"2024-03-04T16:04:00.096922+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:04:00.096922+0000","last_undegraded":"2024-03-04T16:04:00.096922+0000","last_fullsized":"2024-03-04T16:04:00.096922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.421285+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.703150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171188+0000","last_change":"2024-03-04T16:03:58.313071+0000","last_active":"2024-03-04T16:04:00.171188+0000","last_peered":"2024-03-04T16:04:00.171188+0000","last_clean":"2024-03-04T16:04:00.171188+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:04:00.171188+0000","last_undegraded":"2024-03-04T16:04:00.171188+0000","last_fullsized":"2024-03-04T16:04:00.171188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.312925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.312925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.584207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172545+0000","last_change":"2024-03-04T16:02:43.717223+0000","last_active":"2024-03-04T16:04:00.172545+0000","last_peered":"2024-03-04T16:04:00.172545+0000","last_clean":"2024-03-04T16:04:00.172545+0000","last_became_active":"2024-03-04T16:02:43.716945+0000","last_became_peered":"2024-03-04T16:02:43.716945+0000","last_unstale":"2024-03-04T16:04:00.172545+0000","last_undegraded":"2024-03-04T16:04:00.172545+0000","last_fullsized":"2024-03-04T16:04:00.172545+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.985563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170668+0000","last_change":"2024-03-04T16:03:48.260185+0000","last_active":"2024-03-04T16:04:00.170668+0000","last_peered":"2024-03-04T16:04:00.170668+0000","last_clean":"2024-03-04T16:04:00.170668+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:04:00.170668+0000","last_undegraded":"2024-03-04T16:04:00.170668+0000","last_fullsized":"2024-03-04T16:04:00.170668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:48.260047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.260047+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.815204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171810+0000","last_change":"2024-03-04T16:02:47.774009+0000","last_active":"2024-03-04T16:04:00.171810+0000","last_peered":"2024-03-04T16:04:00.171810+0000","last_clean":"2024-03-04T16:04:00.171810+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:04:00.171810+0000","last_undegraded":"2024-03-04T16:04:00.171810+0000","last_fullsized":"2024-03-04T16:04:00.171810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096118+0000","last_change":"2024-03-04T16:02:49.796054+0000","last_active":"2024-03-04T16:04:00.096118+0000","last_peered":"2024-03-04T16:04:00.096118+0000","last_clean":"2024-03-04T16:04:00.096118+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:04:00.096118+0000","last_undegraded":"2024-03-04T16:04:00.096118+0000","last_fullsized":"2024-03-04T16:04:00.096118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.821795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172343+0000","last_change":"2024-03-04T16:03:55.817828+0000","last_active":"2024-03-04T16:04:00.172343+0000","last_peered":"2024-03-04T16:04:00.172343+0000","last_clean":"2024-03-04T16:04:00.172343+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:04:00.172343+0000","last_undegraded":"2024-03-04T16:04:00.172343+0000","last_fullsized":"2024-03-04T16:04:00.172343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:55.817709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.817709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.939046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171220+0000","last_change":"2024-03-04T16:03:48.407212+0000","last_active":"2024-03-04T16:04:00.171220+0000","last_peered":"2024-03-04T16:04:00.171220+0000","last_clean":"2024-03-04T16:04:00.171220+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:04:00.171220+0000","last_undegraded":"2024-03-04T16:04:00.171220+0000","last_fullsized":"2024-03-04T16:04:00.171220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:48.407088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.407088+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:49.238194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095123+0000","last_change":"2024-03-04T16:03:59.048702+0000","last_active":"2024-03-04T16:04:00.095123+0000","last_peered":"2024-03-04T16:04:00.095123+0000","last_clean":"2024-03-04T16:04:00.095123+0000","last_became_active":"2024-03-04T16:02:47.767316+0000","last_became_peered":"2024-03-04T16:02:47.767316+0000","last_unstale":"2024-03-04T16:04:00.095123+0000","last_undegraded":"2024-03-04T16:04:00.095123+0000","last_fullsized":"2024-03-04T16:04:00.095123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:04.396312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170696+0000","last_change":"2024-03-04T16:02:49.797192+0000","last_active":"2024-03-04T16:04:00.170696+0000","last_peered":"2024-03-04T16:04:00.170696+0000","last_clean":"2024-03-04T16:04:00.170696+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:04:00.170696+0000","last_undegraded":"2024-03-04T16:04:00.170696+0000","last_fullsized":"2024-03-04T16:04:00.170696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.448969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095320+0000","last_change":"2024-03-04T16:03:51.984808+0000","last_active":"2024-03-04T16:04:00.095320+0000","last_peered":"2024-03-04T16:04:00.095320+0000","last_clean":"2024-03-04T16:04:00.095320+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:04:00.095320+0000","last_undegraded":"2024-03-04T16:04:00.095320+0000","last_fullsized":"2024-03-04T16:04:00.095320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.984721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.984721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.092310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172295+0000","last_change":"2024-03-04T16:02:43.715496+0000","last_active":"2024-03-04T16:04:00.172295+0000","last_peered":"2024-03-04T16:04:00.172295+0000","last_clean":"2024-03-04T16:04:00.172295+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:04:00.172295+0000","last_undegraded":"2024-03-04T16:04:00.172295+0000","last_fullsized":"2024-03-04T16:04:00.172295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.926799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087201+0000","last_change":"2024-03-04T16:02:45.747239+0000","last_active":"2024-03-04T16:04:00.087201+0000","last_peered":"2024-03-04T16:04:00.087201+0000","last_clean":"2024-03-04T16:04:00.087201+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:04:00.087201+0000","last_undegraded":"2024-03-04T16:04:00.087201+0000","last_fullsized":"2024-03-04T16:04:00.087201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.287130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170394+0000","last_change":"2024-03-04T16:03:50.786884+0000","last_active":"2024-03-04T16:04:00.170394+0000","last_peered":"2024-03-04T16:04:00.170394+0000","last_clean":"2024-03-04T16:04:00.170394+0000","last_became_active":"2024-03-04T16:02:49.793023+0000","last_became_peered":"2024-03-04T16:02:49.793023+0000","last_unstale":"2024-03-04T16:04:00.170394+0000","last_undegraded":"2024-03-04T16:04:00.170394+0000","last_fullsized":"2024-03-04T16:04:00.170394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.786805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.786805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.570842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095255+0000","last_change":"2024-03-04T16:02:47.769558+0000","last_active":"2024-03-04T16:04:00.095255+0000","last_peered":"2024-03-04T16:04:00.095255+0000","last_clean":"2024-03-04T16:04:00.095255+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:04:00.095255+0000","last_undegraded":"2024-03-04T16:04:00.095255+0000","last_fullsized":"2024-03-04T16:04:00.095255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.736215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171855+0000","last_change":"2024-03-04T16:02:43.713788+0000","last_active":"2024-03-04T16:04:00.171855+0000","last_peered":"2024-03-04T16:04:00.171855+0000","last_clean":"2024-03-04T16:04:00.171855+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:04:00.171855+0000","last_undegraded":"2024-03-04T16:04:00.171855+0000","last_fullsized":"2024-03-04T16:04:00.171855+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171391+0000","last_change":"2024-03-04T16:02:45.752568+0000","last_active":"2024-03-04T16:04:00.171391+0000","last_peered":"2024-03-04T16:04:00.171391+0000","last_clean":"2024-03-04T16:04:00.171391+0000","last_became_active":"2024-03-04T16:02:45.752201+0000","last_became_peered":"2024-03-04T16:02:45.752201+0000","last_unstale":"2024-03-04T16:04:00.171391+0000","last_undegraded":"2024-03-04T16:04:00.171391+0000","last_fullsized":"2024-03-04T16:04:00.171391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.531681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096551+0000","last_change":"2024-03-04T16:02:49.793485+0000","last_active":"2024-03-04T16:04:00.096551+0000","last_peered":"2024-03-04T16:04:00.096551+0000","last_clean":"2024-03-04T16:04:00.096551+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:04:00.096551+0000","last_undegraded":"2024-03-04T16:04:00.096551+0000","last_fullsized":"2024-03-04T16:04:00.096551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.099839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170930+0000","last_change":"2024-03-04T16:03:54.344325+0000","last_active":"2024-03-04T16:04:00.170930+0000","last_peered":"2024-03-04T16:04:00.170930+0000","last_clean":"2024-03-04T16:04:00.170930+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:04:00.170930+0000","last_undegraded":"2024-03-04T16:04:00.170930+0000","last_fullsized":"2024-03-04T16:04:00.170930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:03:54.344193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:54.344193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.913538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171536+0000","last_change":"2024-03-04T16:03:59.437627+0000","last_active":"2024-03-04T16:04:00.171536+0000","last_peered":"2024-03-04T16:04:00.171536+0000","last_clean":"2024-03-04T16:04:00.171536+0000","last_became_active":"2024-03-04T16:02:43.717129+0000","last_became_peered":"2024-03-04T16:02:43.717129+0000","last_unstale":"2024-03-04T16:04:00.171536+0000","last_undegraded":"2024-03-04T16:04:00.171536+0000","last_fullsized":"2024-03-04T16:04:00.171536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.437524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.437524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.171391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":115,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171435+0000","last_change":"2024-03-04T16:02:45.753468+0000","last_active":"2024-03-04T16:04:00.171435+0000","last_peered":"2024-03-04T16:04:00.171435+0000","last_clean":"2024-03-04T16:04:00.171435+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:04:00.171435+0000","last_undegraded":"2024-03-04T16:04:00.171435+0000","last_fullsized":"2024-03-04T16:04:00.171435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.486508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171374+0000","last_change":"2024-03-04T16:02:47.773975+0000","last_active":"2024-03-04T16:04:00.171374+0000","last_peered":"2024-03-04T16:04:00.171374+0000","last_clean":"2024-03-04T16:04:00.171374+0000","last_became_active":"2024-03-04T16:02:47.773547+0000","last_became_peered":"2024-03-04T16:02:47.773547+0000","last_unstale":"2024-03-04T16:04:00.171374+0000","last_undegraded":"2024-03-04T16:04:00.171374+0000","last_fullsized":"2024-03-04T16:04:00.171374+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172366+0000","last_change":"2024-03-04T16:02:49.791079+0000","last_active":"2024-03-04T16:04:00.172366+0000","last_peered":"2024-03-04T16:04:00.172366+0000","last_clean":"2024-03-04T16:04:00.172366+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:04:00.172366+0000","last_undegraded":"2024-03-04T16:04:00.172366+0000","last_fullsized":"2024-03-04T16:04:00.172366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170995+0000","last_change":"2024-03-04T16:02:43.714877+0000","last_active":"2024-03-04T16:04:00.170995+0000","last_peered":"2024-03-04T16:04:00.170995+0000","last_clean":"2024-03-04T16:04:00.170995+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:04:00.170995+0000","last_undegraded":"2024-03-04T16:04:00.170995+0000","last_fullsized":"2024-03-04T16:04:00.170995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095150+0000","last_change":"2024-03-04T16:03:47.979184+0000","last_active":"2024-03-04T16:04:00.095150+0000","last_peered":"2024-03-04T16:04:00.095150+0000","last_clean":"2024-03-04T16:04:00.095150+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:04:00.095150+0000","last_undegraded":"2024-03-04T16:04:00.095150+0000","last_fullsized":"2024-03-04T16:04:00.095150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.979058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.979058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.315882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171982+0000","last_change":"2024-03-04T16:02:45.748924+0000","last_active":"2024-03-04T16:04:00.171982+0000","last_peered":"2024-03-04T16:04:00.171982+0000","last_clean":"2024-03-04T16:04:00.171982+0000","last_became_active":"2024-03-04T16:02:45.748615+0000","last_became_peered":"2024-03-04T16:02:45.748615+0000","last_unstale":"2024-03-04T16:04:00.171982+0000","last_undegraded":"2024-03-04T16:04:00.171982+0000","last_fullsized":"2024-03-04T16:04:00.171982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.429848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095472+0000","last_change":"2024-03-04T16:02:47.769657+0000","last_active":"2024-03-04T16:04:00.095472+0000","last_peered":"2024-03-04T16:04:00.095472+0000","last_clean":"2024-03-04T16:04:00.095472+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:04:00.095472+0000","last_undegraded":"2024-03-04T16:04:00.095472+0000","last_fullsized":"2024-03-04T16:04:00.095472+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172223+0000","last_change":"2024-03-04T16:02:49.794677+0000","last_active":"2024-03-04T16:04:00.172223+0000","last_peered":"2024-03-04T16:04:00.172223+0000","last_clean":"2024-03-04T16:04:00.172223+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:04:00.172223+0000","last_undegraded":"2024-03-04T16:04:00.172223+0000","last_fullsized":"2024-03-04T16:04:00.172223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.070330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170921+0000","last_change":"2024-03-04T16:03:12.851794+0000","last_active":"2024-03-04T16:04:00.170921+0000","last_peered":"2024-03-04T16:04:00.170921+0000","last_clean":"2024-03-04T16:04:00.170921+0000","last_became_active":"2024-03-04T16:02:34.079164+0000","last_became_peered":"2024-03-04T16:02:34.079164+0000","last_unstale":"2024-03-04T16:04:00.170921+0000","last_undegraded":"2024-03-04T16:04:00.170921+0000","last_fullsized":"2024-03-04T16:04:00.170921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'119","last_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:03:12.851677+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:36.356853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096256+0000","last_change":"2024-03-04T16:03:57.445763+0000","last_active":"2024-03-04T16:04:00.096256+0000","last_peered":"2024-03-04T16:04:00.096256+0000","last_clean":"2024-03-04T16:04:00.096256+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:04:00.096256+0000","last_undegraded":"2024-03-04T16:04:00.096256+0000","last_fullsized":"2024-03-04T16:04:00.096256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.445659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.445659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.402395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":97,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087537+0000","last_change":"2024-03-04T16:03:58.764928+0000","last_active":"2024-03-04T16:04:00.087537+0000","last_peered":"2024-03-04T16:04:00.087537+0000","last_clean":"2024-03-04T16:04:00.087537+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:04:00.087537+0000","last_undegraded":"2024-03-04T16:04:00.087537+0000","last_fullsized":"2024-03-04T16:04:00.087537+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:03:58.764821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.764821+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.352281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172055+0000","last_change":"2024-03-04T16:03:52.917188+0000","last_active":"2024-03-04T16:04:00.172055+0000","last_peered":"2024-03-04T16:04:00.172055+0000","last_clean":"2024-03-04T16:04:00.172055+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:04:00.172055+0000","last_undegraded":"2024-03-04T16:04:00.172055+0000","last_fullsized":"2024-03-04T16:04:00.172055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.917090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.917090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:00.018319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171270+0000","last_change":"2024-03-04T16:03:52.944487+0000","last_active":"2024-03-04T16:04:00.171270+0000","last_peered":"2024-03-04T16:04:00.171270+0000","last_clean":"2024-03-04T16:04:00.171270+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:04:00.171270+0000","last_undegraded":"2024-03-04T16:04:00.171270+0000","last_fullsized":"2024-03-04T16:04:00.171270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.944410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.132708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170566+0000","last_change":"2024-03-04T16:02:43.713670+0000","last_active":"2024-03-04T16:04:00.170566+0000","last_peered":"2024-03-04T16:04:00.170566+0000","last_clean":"2024-03-04T16:04:00.170566+0000","last_became_active":"2024-03-04T16:02:43.713276+0000","last_became_peered":"2024-03-04T16:02:43.713276+0000","last_unstale":"2024-03-04T16:04:00.170566+0000","last_undegraded":"2024-03-04T16:04:00.170566+0000","last_fullsized":"2024-03-04T16:04:00.170566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.212292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":140,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:01.129643+0000","last_change":"2024-03-04T16:02:46.749606+0000","last_active":"2024-03-04T16:04:01.129643+0000","last_peered":"2024-03-04T16:04:01.129643+0000","last_clean":"2024-03-04T16:04:01.129643+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:04:01.129643+0000","last_undegraded":"2024-03-04T16:04:01.129643+0000","last_fullsized":"2024-03-04T16:04:01.129643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:02:41.630915+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171663+0000","last_change":"2024-03-04T16:02:45.755785+0000","last_active":"2024-03-04T16:04:00.171663+0000","last_peered":"2024-03-04T16:04:00.171663+0000","last_clean":"2024-03-04T16:04:00.171663+0000","last_became_active":"2024-03-04T16:02:45.755435+0000","last_became_peered":"2024-03-04T16:02:45.755435+0000","last_unstale":"2024-03-04T16:04:00.171663+0000","last_undegraded":"2024-03-04T16:04:00.171663+0000","last_fullsized":"2024-03-04T16:04:00.171663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.974009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172637+0000","last_change":"2024-03-04T16:03:57.296517+0000","last_active":"2024-03-04T16:04:00.172637+0000","last_peered":"2024-03-04T16:04:00.172637+0000","last_clean":"2024-03-04T16:04:00.172637+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:04:00.172637+0000","last_undegraded":"2024-03-04T16:04:00.172637+0000","last_fullsized":"2024-03-04T16:04:00.172637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.296421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.296421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.272283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170706+0000","last_change":"2024-03-04T16:02:49.785263+0000","last_active":"2024-03-04T16:04:00.170706+0000","last_peered":"2024-03-04T16:04:00.170706+0000","last_clean":"2024-03-04T16:04:00.170706+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:04:00.170706+0000","last_undegraded":"2024-03-04T16:04:00.170706+0000","last_fullsized":"2024-03-04T16:04:00.170706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171392+0000","last_change":"2024-03-04T16:02:43.719367+0000","last_active":"2024-03-04T16:04:00.171392+0000","last_peered":"2024-03-04T16:04:00.171392+0000","last_clean":"2024-03-04T16:04:00.171392+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:04:00.171392+0000","last_undegraded":"2024-03-04T16:04:00.171392+0000","last_fullsized":"2024-03-04T16:04:00.171392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171241+0000","last_change":"2024-03-04T16:02:45.744516+0000","last_active":"2024-03-04T16:04:00.171241+0000","last_peered":"2024-03-04T16:04:00.171241+0000","last_clean":"2024-03-04T16:04:00.171241+0000","last_became_active":"2024-03-04T16:02:45.744172+0000","last_became_peered":"2024-03-04T16:02:45.744172+0000","last_unstale":"2024-03-04T16:04:00.171241+0000","last_undegraded":"2024-03-04T16:04:00.171241+0000","last_fullsized":"2024-03-04T16:04:00.171241+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.725177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087204+0000","last_change":"2024-03-04T16:03:50.770141+0000","last_active":"2024-03-04T16:04:00.087204+0000","last_peered":"2024-03-04T16:04:00.087204+0000","last_clean":"2024-03-04T16:04:00.087204+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:04:00.087204+0000","last_undegraded":"2024-03-04T16:04:00.087204+0000","last_fullsized":"2024-03-04T16:04:00.087204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.770048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.770048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.417482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170347+0000","last_change":"2024-03-04T16:02:49.797759+0000","last_active":"2024-03-04T16:04:00.170347+0000","last_peered":"2024-03-04T16:04:00.170347+0000","last_clean":"2024-03-04T16:04:00.170347+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:04:00.170347+0000","last_undegraded":"2024-03-04T16:04:00.170347+0000","last_fullsized":"2024-03-04T16:04:00.170347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.105311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172561+0000","last_change":"2024-03-04T16:02:43.711336+0000","last_active":"2024-03-04T16:04:00.172561+0000","last_peered":"2024-03-04T16:04:00.172561+0000","last_clean":"2024-03-04T16:04:00.172561+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:04:00.172561+0000","last_undegraded":"2024-03-04T16:04:00.172561+0000","last_fullsized":"2024-03-04T16:04:00.172561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:11.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170915+0000","last_change":"2024-03-04T16:02:45.748912+0000","last_active":"2024-03-04T16:04:00.170915+0000","last_peered":"2024-03-04T16:04:00.170915+0000","last_clean":"2024-03-04T16:04:00.170915+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:04:00.170915+0000","last_undegraded":"2024-03-04T16:04:00.170915+0000","last_fullsized":"2024-03-04T16:04:00.170915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171006+0000","last_change":"2024-03-04T16:02:47.777918+0000","last_active":"2024-03-04T16:04:00.171006+0000","last_peered":"2024-03-04T16:04:00.171006+0000","last_clean":"2024-03-04T16:04:00.171006+0000","last_became_active":"2024-03-04T16:02:47.777462+0000","last_became_peered":"2024-03-04T16:02:47.777462+0000","last_unstale":"2024-03-04T16:04:00.171006+0000","last_undegraded":"2024-03-04T16:04:00.171006+0000","last_fullsized":"2024-03-04T16:04:00.171006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.300423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171829+0000","last_change":"2024-03-04T16:03:56.858487+0000","last_active":"2024-03-04T16:04:00.171829+0000","last_peered":"2024-03-04T16:04:00.171829+0000","last_clean":"2024-03-04T16:04:00.171829+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:04:00.171829+0000","last_undegraded":"2024-03-04T16:04:00.171829+0000","last_fullsized":"2024-03-04T16:04:00.171829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.858376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.858376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.769864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171668+0000","last_change":"2024-03-04T16:02:43.712924+0000","last_active":"2024-03-04T16:04:00.171668+0000","last_peered":"2024-03-04T16:04:00.171668+0000","last_clean":"2024-03-04T16:04:00.171668+0000","last_became_active":"2024-03-04T16:02:43.712287+0000","last_became_peered":"2024-03-04T16:02:43.712287+0000","last_unstale":"2024-03-04T16:04:00.171668+0000","last_undegraded":"2024-03-04T16:04:00.171668+0000","last_fullsized":"2024-03-04T16:04:00.171668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.101366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"75'14","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171072+0000","last_change":"2024-03-04T16:02:45.739793+0000","last_active":"2024-03-04T16:04:00.171072+0000","last_peered":"2024-03-04T16:04:00.171072+0000","last_clean":"2024-03-04T16:04:00.171072+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:04:00.171072+0000","last_undegraded":"2024-03-04T16:04:00.171072+0000","last_fullsized":"2024-03-04T16:04:00.171072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096078+0000","last_change":"2024-03-04T16:02:47.767792+0000","last_active":"2024-03-04T16:04:00.096078+0000","last_peered":"2024-03-04T16:04:00.096078+0000","last_clean":"2024-03-04T16:04:00.096078+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:04:00.096078+0000","last_undegraded":"2024-03-04T16:04:00.096078+0000","last_fullsized":"2024-03-04T16:04:00.096078+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171503+0000","last_change":"2024-03-04T16:02:49.789412+0000","last_active":"2024-03-04T16:04:00.171503+0000","last_peered":"2024-03-04T16:04:00.171503+0000","last_clean":"2024-03-04T16:04:00.171503+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:04:00.171503+0000","last_undegraded":"2024-03-04T16:04:00.171503+0000","last_fullsized":"2024-03-04T16:04:00.171503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.799791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096671+0000","last_change":"2024-03-04T16:02:43.716171+0000","last_active":"2024-03-04T16:04:00.096671+0000","last_peered":"2024-03-04T16:04:00.096671+0000","last_clean":"2024-03-04T16:04:00.096671+0000","last_became_active":"2024-03-04T16:02:43.715346+0000","last_became_peered":"2024-03-04T16:02:43.715346+0000","last_unstale":"2024-03-04T16:04:00.096671+0000","last_undegraded":"2024-03-04T16:04:00.096671+0000","last_fullsized":"2024-03-04T16:04:00.096671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095990+0000","last_change":"2024-03-04T16:02:45.749977+0000","last_active":"2024-03-04T16:04:00.095990+0000","last_peered":"2024-03-04T16:04:00.095990+0000","last_clean":"2024-03-04T16:04:00.095990+0000","last_became_active":"2024-03-04T16:02:45.749633+0000","last_became_peered":"2024-03-04T16:02:45.749633+0000","last_unstale":"2024-03-04T16:04:00.095990+0000","last_undegraded":"2024-03-04T16:04:00.095990+0000","last_fullsized":"2024-03-04T16:04:00.095990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.646959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170756+0000","last_change":"2024-03-04T16:02:47.768154+0000","last_active":"2024-03-04T16:04:00.170756+0000","last_peered":"2024-03-04T16:04:00.170756+0000","last_clean":"2024-03-04T16:04:00.170756+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:04:00.170756+0000","last_undegraded":"2024-03-04T16:04:00.170756+0000","last_fullsized":"2024-03-04T16:04:00.170756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.581920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172282+0000","last_change":"2024-03-04T16:02:49.792150+0000","last_active":"2024-03-04T16:04:00.172282+0000","last_peered":"2024-03-04T16:04:00.172282+0000","last_clean":"2024-03-04T16:04:00.172282+0000","last_became_active":"2024-03-04T16:02:49.791851+0000","last_became_peered":"2024-03-04T16:02:49.791851+0000","last_unstale":"2024-03-04T16:04:00.172282+0000","last_undegraded":"2024-03-04T16:04:00.172282+0000","last_fullsized":"2024-03-04T16:04:00.172282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.262236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.173031+0000","last_change":"2024-03-04T16:02:47.768859+0000","last_active":"2024-03-04T16:04:00.173031+0000","last_peered":"2024-03-04T16:04:00.173031+0000","last_clean":"2024-03-04T16:04:00.173031+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:04:00.173031+0000","last_undegraded":"2024-03-04T16:04:00.173031+0000","last_fullsized":"2024-03-04T16:04:00.173031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.459494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095980+0000","last_change":"2024-03-04T16:03:58.462106+0000","last_active":"2024-03-04T16:04:00.095980+0000","last_peered":"2024-03-04T16:04:00.095980+0000","last_clean":"2024-03-04T16:04:00.095980+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:04:00.095980+0000","last_undegraded":"2024-03-04T16:04:00.095980+0000","last_fullsized":"2024-03-04T16:04:00.095980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.462018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:08.477439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171988+0000","last_change":"2024-03-04T16:03:46.963168+0000","last_active":"2024-03-04T16:04:00.171988+0000","last_peered":"2024-03-04T16:04:00.171988+0000","last_clean":"2024-03-04T16:04:00.171988+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:04:00.171988+0000","last_undegraded":"2024-03-04T16:04:00.171988+0000","last_fullsized":"2024-03-04T16:04:00.171988+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:46.963065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.963065+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.801631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096277+0000","last_change":"2024-03-04T16:02:49.794003+0000","last_active":"2024-03-04T16:04:00.096277+0000","last_peered":"2024-03-04T16:04:00.096277+0000","last_clean":"2024-03-04T16:04:00.096277+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:04:00.096277+0000","last_undegraded":"2024-03-04T16:04:00.096277+0000","last_fullsized":"2024-03-04T16:04:00.096277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.636557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172548+0000","last_change":"2024-03-04T16:02:47.766009+0000","last_active":"2024-03-04T16:04:00.172548+0000","last_peered":"2024-03-04T16:04:00.172548+0000","last_clean":"2024-03-04T16:04:00.172548+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:04:00.172548+0000","last_undegraded":"2024-03-04T16:04:00.172548+0000","last_fullsized":"2024-03-04T16:04:00.172548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.260597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170870+0000","last_change":"2024-03-04T16:02:43.705402+0000","last_active":"2024-03-04T16:04:00.170870+0000","last_peered":"2024-03-04T16:04:00.170870+0000","last_clean":"2024-03-04T16:04:00.170870+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:04:00.170870+0000","last_undegraded":"2024-03-04T16:04:00.170870+0000","last_fullsized":"2024-03-04T16:04:00.170870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.029120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172915+0000","last_change":"2024-03-04T16:02:45.746393+0000","last_active":"2024-03-04T16:04:00.172915+0000","last_peered":"2024-03-04T16:04:00.172915+0000","last_clean":"2024-03-04T16:04:00.172915+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:04:00.172915+0000","last_undegraded":"2024-03-04T16:04:00.172915+0000","last_fullsized":"2024-03-04T16:04:00.172915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.554587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095604+0000","last_change":"2024-03-04T16:02:49.789192+0000","last_active":"2024-03-04T16:04:00.095604+0000","last_peered":"2024-03-04T16:04:00.095604+0000","last_clean":"2024-03-04T16:04:00.095604+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:04:00.095604+0000","last_undegraded":"2024-03-04T16:04:00.095604+0000","last_fullsized":"2024-03-04T16:04:00.095604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.114265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171530+0000","last_change":"2024-03-04T16:02:47.768568+0000","last_active":"2024-03-04T16:04:00.171530+0000","last_peered":"2024-03-04T16:04:00.171530+0000","last_clean":"2024-03-04T16:04:00.171530+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:04:00.171530+0000","last_undegraded":"2024-03-04T16:04:00.171530+0000","last_fullsized":"2024-03-04T16:04:00.171530+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087321+0000","last_change":"2024-03-04T16:02:43.716227+0000","last_active":"2024-03-04T16:04:00.087321+0000","last_peered":"2024-03-04T16:04:00.087321+0000","last_clean":"2024-03-04T16:04:00.087321+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:04:00.087321+0000","last_undegraded":"2024-03-04T16:04:00.087321+0000","last_fullsized":"2024-03-04T16:04:00.087321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172875+0000","last_change":"2024-03-04T16:02:45.743746+0000","last_active":"2024-03-04T16:04:00.172875+0000","last_peered":"2024-03-04T16:04:00.172875+0000","last_clean":"2024-03-04T16:04:00.172875+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:04:00.172875+0000","last_undegraded":"2024-03-04T16:04:00.172875+0000","last_fullsized":"2024-03-04T16:04:00.172875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.769608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.173040+0000","last_change":"2024-03-04T16:02:49.785271+0000","last_active":"2024-03-04T16:04:00.173040+0000","last_peered":"2024-03-04T16:04:00.173040+0000","last_clean":"2024-03-04T16:04:00.173040+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:04:00.173040+0000","last_undegraded":"2024-03-04T16:04:00.173040+0000","last_fullsized":"2024-03-04T16:04:00.173040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095409+0000","last_change":"2024-03-04T16:02:47.770597+0000","last_active":"2024-03-04T16:04:00.095409+0000","last_peered":"2024-03-04T16:04:00.095409+0000","last_clean":"2024-03-04T16:04:00.095409+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:04:00.095409+0000","last_undegraded":"2024-03-04T16:04:00.095409+0000","last_fullsized":"2024-03-04T16:04:00.095409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.503152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096393+0000","last_change":"2024-03-04T16:02:43.715105+0000","last_active":"2024-03-04T16:04:00.096393+0000","last_peered":"2024-03-04T16:04:00.096393+0000","last_clean":"2024-03-04T16:04:00.096393+0000","last_became_active":"2024-03-04T16:02:43.714792+0000","last_became_peered":"2024-03-04T16:02:43.714792+0000","last_unstale":"2024-03-04T16:04:00.096393+0000","last_undegraded":"2024-03-04T16:04:00.096393+0000","last_fullsized":"2024-03-04T16:04:00.096393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171318+0000","last_change":"2024-03-04T16:02:45.744111+0000","last_active":"2024-03-04T16:04:00.171318+0000","last_peered":"2024-03-04T16:04:00.171318+0000","last_clean":"2024-03-04T16:04:00.171318+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:04:00.171318+0000","last_undegraded":"2024-03-04T16:04:00.171318+0000","last_fullsized":"2024-03-04T16:04:00.171318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.765686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":400,"ondisk_log_size":400,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070484,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32560,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712592,"statfs":{"total":95995035648,"available":95961694208,"internally_reserved":0,"allocated":1413120,"data_stored":946996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364824,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1388544,"data_stored":944684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":856064,"data_stored":356088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28400,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716752,"statfs":{"total":95995035648,"available":95965954048,"internally_reserved":0,"allocated":1413120,"data_stored":946340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":790528,"data_stored":354816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32000,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713152,"statfs":{"total":95995035648,"available":95962267648,"internally_reserved":0,"allocated":835584,"data_stored":356082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953506,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":823296,"data_stored":355306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626462,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":798720,"data_stored":354642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:04:05.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:04:05.473 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:05 smithi176 ceph-mon[33619]: Deploying daemon grafana.a on smithi176 2024-03-04T16:04:05.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:04:05.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[38113]: Deploying daemon grafana.a on smithi176 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:05.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:05 smithi038 ceph-mon[29779]: Deploying daemon grafana.a on smithi176 2024-03-04T16:04:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:06.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:06.748 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:06.749 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:06 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:06.429Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000935456s 2024-03-04T16:04:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:04:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:04:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:06 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:07.709 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:07.711 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:04:07.847 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 3.c deep-scrub starts 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 3.c deep-scrub ok 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 4.1 deep-scrub starts 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[29779]: 4.1 deep-scrub ok 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 3.c deep-scrub starts 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 3.c deep-scrub ok 2024-03-04T16:04:07.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 4.1 deep-scrub starts 2024-03-04T16:04:07.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:07 smithi038 ceph-mon[38113]: 4.1 deep-scrub ok 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 3.c deep-scrub starts 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 3.c deep-scrub ok 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 4.1 deep-scrub starts 2024-03-04T16:04:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:07 smithi176 ceph-mon[33619]: 4.1 deep-scrub ok 2024-03-04T16:04:08.469 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-03-04T16:04:07.291215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1012,"num_read_kb":858,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240892,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720324,"statfs":{"total":767960285184,"available":767713611776,"internally_reserved":0,"allocated":8318976,"data_stored":4614954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002287"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171404+0000","last_change":"2024-03-04T16:03:57.477127+0000","last_active":"2024-03-04T16:04:00.171404+0000","last_peered":"2024-03-04T16:04:00.171404+0000","last_clean":"2024-03-04T16:04:00.171404+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:04:00.171404+0000","last_undegraded":"2024-03-04T16:04:00.171404+0000","last_fullsized":"2024-03-04T16:04:00.171404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.477014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.477014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.118199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170834+0000","last_change":"2024-03-04T16:02:47.767863+0000","last_active":"2024-03-04T16:04:00.170834+0000","last_peered":"2024-03-04T16:04:00.170834+0000","last_clean":"2024-03-04T16:04:00.170834+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:04:00.170834+0000","last_undegraded":"2024-03-04T16:04:00.170834+0000","last_fullsized":"2024-03-04T16:04:00.170834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.395983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171558+0000","last_change":"2024-03-04T16:03:45.773330+0000","last_active":"2024-03-04T16:04:00.171558+0000","last_peered":"2024-03-04T16:04:00.171558+0000","last_clean":"2024-03-04T16:04:00.171558+0000","last_became_active":"2024-03-04T16:02:43.714846+0000","last_became_peered":"2024-03-04T16:02:43.714846+0000","last_unstale":"2024-03-04T16:04:00.171558+0000","last_undegraded":"2024-03-04T16:04:00.171558+0000","last_fullsized":"2024-03-04T16:04:00.171558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:45.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.773213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.488104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170701+0000","last_change":"2024-03-04T16:02:45.748421+0000","last_active":"2024-03-04T16:04:00.170701+0000","last_peered":"2024-03-04T16:04:00.170701+0000","last_clean":"2024-03-04T16:04:00.170701+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:04:00.170701+0000","last_undegraded":"2024-03-04T16:04:00.170701+0000","last_fullsized":"2024-03-04T16:04:00.170701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.620264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.380440+0000","last_change":"2024-03-04T16:04:05.380482+0000","last_active":"2024-03-04T16:04:05.380440+0000","last_peered":"2024-03-04T16:04:05.380440+0000","last_clean":"2024-03-04T16:04:05.380440+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:04:05.380440+0000","last_undegraded":"2024-03-04T16:04:05.380440+0000","last_fullsized":"2024-03-04T16:04:05.380440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.380364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.380364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.798710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172064+0000","last_change":"2024-03-04T16:02:47.766441+0000","last_active":"2024-03-04T16:04:00.172064+0000","last_peered":"2024-03-04T16:04:00.172064+0000","last_clean":"2024-03-04T16:04:00.172064+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:04:00.172064+0000","last_undegraded":"2024-03-04T16:04:00.172064+0000","last_fullsized":"2024-03-04T16:04:00.172064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096811+0000","last_change":"2024-03-04T16:02:43.712774+0000","last_active":"2024-03-04T16:04:00.096811+0000","last_peered":"2024-03-04T16:04:00.096811+0000","last_clean":"2024-03-04T16:04:00.096811+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:04:00.096811+0000","last_undegraded":"2024-03-04T16:04:00.096811+0000","last_fullsized":"2024-03-04T16:04:00.096811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.121882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096197+0000","last_change":"2024-03-04T16:03:50.379226+0000","last_active":"2024-03-04T16:04:00.096197+0000","last_peered":"2024-03-04T16:04:00.096197+0000","last_clean":"2024-03-04T16:04:00.096197+0000","last_became_active":"2024-03-04T16:02:45.749383+0000","last_became_peered":"2024-03-04T16:02:45.749383+0000","last_unstale":"2024-03-04T16:04:00.096197+0000","last_undegraded":"2024-03-04T16:04:00.096197+0000","last_fullsized":"2024-03-04T16:04:00.096197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:50.379077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.379077+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.153393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170489+0000","last_change":"2024-03-04T16:03:56.851189+0000","last_active":"2024-03-04T16:04:00.170489+0000","last_peered":"2024-03-04T16:04:00.170489+0000","last_clean":"2024-03-04T16:04:00.170489+0000","last_became_active":"2024-03-04T16:02:49.791142+0000","last_became_peered":"2024-03-04T16:02:49.791142+0000","last_unstale":"2024-03-04T16:04:00.170489+0000","last_undegraded":"2024-03-04T16:04:00.170489+0000","last_fullsized":"2024-03-04T16:04:00.170489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.851094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.851094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.117129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171309+0000","last_change":"2024-03-04T16:02:47.775367+0000","last_active":"2024-03-04T16:04:00.171309+0000","last_peered":"2024-03-04T16:04:00.171309+0000","last_clean":"2024-03-04T16:04:00.171309+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:04:00.171309+0000","last_undegraded":"2024-03-04T16:04:00.171309+0000","last_fullsized":"2024-03-04T16:04:00.171309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.508809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172396+0000","last_change":"2024-03-04T16:03:51.290213+0000","last_active":"2024-03-04T16:04:00.172396+0000","last_peered":"2024-03-04T16:04:00.172396+0000","last_clean":"2024-03-04T16:04:00.172396+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:04:00.172396+0000","last_undegraded":"2024-03-04T16:04:00.172396+0000","last_fullsized":"2024-03-04T16:04:00.172396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.290117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.290117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.885059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.434104+0000","last_change":"2024-03-04T16:04:00.434148+0000","last_active":"2024-03-04T16:04:00.434104+0000","last_peered":"2024-03-04T16:04:00.434104+0000","last_clean":"2024-03-04T16:04:00.434104+0000","last_became_active":"2024-03-04T16:02:45.745532+0000","last_became_peered":"2024-03-04T16:02:45.745532+0000","last_unstale":"2024-03-04T16:04:00.434104+0000","last_undegraded":"2024-03-04T16:04:00.434104+0000","last_fullsized":"2024-03-04T16:04:00.434104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.434035+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.035919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:02.311200+0000","last_change":"2024-03-04T16:04:02.311228+0000","last_active":"2024-03-04T16:04:02.311200+0000","last_peered":"2024-03-04T16:04:02.311200+0000","last_clean":"2024-03-04T16:04:02.311200+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:04:02.311200+0000","last_undegraded":"2024-03-04T16:04:02.311200+0000","last_fullsized":"2024-03-04T16:04:02.311200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:02.311103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:02.311103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:06.585200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171213+0000","last_change":"2024-03-04T16:02:47.768573+0000","last_active":"2024-03-04T16:04:00.171213+0000","last_peered":"2024-03-04T16:04:00.171213+0000","last_clean":"2024-03-04T16:04:00.171213+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:04:00.171213+0000","last_undegraded":"2024-03-04T16:04:00.171213+0000","last_fullsized":"2024-03-04T16:04:00.171213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.292580+0000","last_change":"2024-03-04T16:04:00.292627+0000","last_active":"2024-03-04T16:04:00.292580+0000","last_peered":"2024-03-04T16:04:00.292580+0000","last_clean":"2024-03-04T16:04:00.292580+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:04:00.292580+0000","last_undegraded":"2024-03-04T16:04:00.292580+0000","last_fullsized":"2024-03-04T16:04:00.292580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:00.292499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.292499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.161517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171851+0000","last_change":"2024-03-04T16:02:45.748750+0000","last_active":"2024-03-04T16:04:00.171851+0000","last_peered":"2024-03-04T16:04:00.171851+0000","last_clean":"2024-03-04T16:04:00.171851+0000","last_became_active":"2024-03-04T16:02:45.747787+0000","last_became_peered":"2024-03-04T16:02:45.747787+0000","last_unstale":"2024-03-04T16:04:00.171851+0000","last_undegraded":"2024-03-04T16:04:00.171851+0000","last_fullsized":"2024-03-04T16:04:00.171851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:14.663457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172933+0000","last_change":"2024-03-04T16:02:49.785706+0000","last_active":"2024-03-04T16:04:00.172933+0000","last_peered":"2024-03-04T16:04:00.172933+0000","last_clean":"2024-03-04T16:04:00.172933+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:04:00.172933+0000","last_undegraded":"2024-03-04T16:04:00.172933+0000","last_fullsized":"2024-03-04T16:04:00.172933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.708627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171854+0000","last_change":"2024-03-04T16:02:47.768091+0000","last_active":"2024-03-04T16:04:00.171854+0000","last_peered":"2024-03-04T16:04:00.171854+0000","last_clean":"2024-03-04T16:04:00.171854+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:04:00.171854+0000","last_undegraded":"2024-03-04T16:04:00.171854+0000","last_fullsized":"2024-03-04T16:04:00.171854+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171158+0000","last_change":"2024-03-04T16:02:43.714700+0000","last_active":"2024-03-04T16:04:00.171158+0000","last_peered":"2024-03-04T16:04:00.171158+0000","last_clean":"2024-03-04T16:04:00.171158+0000","last_became_active":"2024-03-04T16:02:43.714419+0000","last_became_peered":"2024-03-04T16:02:43.714419+0000","last_unstale":"2024-03-04T16:04:00.171158+0000","last_undegraded":"2024-03-04T16:04:00.171158+0000","last_fullsized":"2024-03-04T16:04:00.171158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.557877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095293+0000","last_change":"2024-03-04T16:03:45.980725+0000","last_active":"2024-03-04T16:04:00.095293+0000","last_peered":"2024-03-04T16:04:00.095293+0000","last_clean":"2024-03-04T16:04:00.095293+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:04:00.095293+0000","last_undegraded":"2024-03-04T16:04:00.095293+0000","last_fullsized":"2024-03-04T16:04:00.095293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:45.980606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.980606+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:01.702710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096882+0000","last_change":"2024-03-04T16:02:49.794035+0000","last_active":"2024-03-04T16:04:00.096882+0000","last_peered":"2024-03-04T16:04:00.096882+0000","last_clean":"2024-03-04T16:04:00.096882+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:04:00.096882+0000","last_undegraded":"2024-03-04T16:04:00.096882+0000","last_fullsized":"2024-03-04T16:04:00.096882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.379614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171229+0000","last_change":"2024-03-04T16:02:47.769883+0000","last_active":"2024-03-04T16:04:00.171229+0000","last_peered":"2024-03-04T16:04:00.171229+0000","last_clean":"2024-03-04T16:04:00.171229+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:04:00.171229+0000","last_undegraded":"2024-03-04T16:04:00.171229+0000","last_fullsized":"2024-03-04T16:04:00.171229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.833777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171760+0000","last_change":"2024-03-04T16:03:46.423866+0000","last_active":"2024-03-04T16:04:00.171760+0000","last_peered":"2024-03-04T16:04:00.171760+0000","last_clean":"2024-03-04T16:04:00.171760+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:04:00.171760+0000","last_undegraded":"2024-03-04T16:04:00.171760+0000","last_fullsized":"2024-03-04T16:04:00.171760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.423767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.868075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096446+0000","last_change":"2024-03-04T16:03:55.411658+0000","last_active":"2024-03-04T16:04:00.096446+0000","last_peered":"2024-03-04T16:04:00.096446+0000","last_clean":"2024-03-04T16:04:00.096446+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:04:00.096446+0000","last_undegraded":"2024-03-04T16:04:00.096446+0000","last_fullsized":"2024-03-04T16:04:00.096446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:55.411554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.411554+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.742258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171968+0000","last_change":"2024-03-04T16:02:49.792726+0000","last_active":"2024-03-04T16:04:00.171968+0000","last_peered":"2024-03-04T16:04:00.171968+0000","last_clean":"2024-03-04T16:04:00.171968+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:04:00.171968+0000","last_undegraded":"2024-03-04T16:04:00.171968+0000","last_fullsized":"2024-03-04T16:04:00.171968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.657727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087706+0000","last_change":"2024-03-04T16:02:47.772339+0000","last_active":"2024-03-04T16:04:00.087706+0000","last_peered":"2024-03-04T16:04:00.087706+0000","last_clean":"2024-03-04T16:04:00.087706+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:04:00.087706+0000","last_undegraded":"2024-03-04T16:04:00.087706+0000","last_fullsized":"2024-03-04T16:04:00.087706+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.633218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:01.014974+0000","last_change":"2024-03-04T16:04:01.015009+0000","last_active":"2024-03-04T16:04:01.014974+0000","last_peered":"2024-03-04T16:04:01.014974+0000","last_clean":"2024-03-04T16:04:01.014974+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:04:01.014974+0000","last_undegraded":"2024-03-04T16:04:01.014974+0000","last_fullsized":"2024-03-04T16:04:01.014974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:01.014871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:01.014871+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:13.772804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171781+0000","last_change":"2024-03-04T16:02:45.745063+0000","last_active":"2024-03-04T16:04:00.171781+0000","last_peered":"2024-03-04T16:04:00.171781+0000","last_clean":"2024-03-04T16:04:00.171781+0000","last_became_active":"2024-03-04T16:02:45.744705+0000","last_became_peered":"2024-03-04T16:02:45.744705+0000","last_unstale":"2024-03-04T16:04:00.171781+0000","last_undegraded":"2024-03-04T16:04:00.171781+0000","last_fullsized":"2024-03-04T16:04:00.171781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.665716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171960+0000","last_change":"2024-03-04T16:03:53.955279+0000","last_active":"2024-03-04T16:04:00.171960+0000","last_peered":"2024-03-04T16:04:00.171960+0000","last_clean":"2024-03-04T16:04:00.171960+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:04:00.171960+0000","last_undegraded":"2024-03-04T16:04:00.171960+0000","last_fullsized":"2024-03-04T16:04:00.171960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:53.955186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:53.955186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.764558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171905+0000","last_change":"2024-03-04T16:03:57.874266+0000","last_active":"2024-03-04T16:04:00.171905+0000","last_peered":"2024-03-04T16:04:00.171905+0000","last_clean":"2024-03-04T16:04:00.171905+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:04:00.171905+0000","last_undegraded":"2024-03-04T16:04:00.171905+0000","last_fullsized":"2024-03-04T16:04:00.171905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.874150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.874150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.705755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095851+0000","last_change":"2024-03-04T16:03:47.354437+0000","last_active":"2024-03-04T16:04:00.095851+0000","last_peered":"2024-03-04T16:04:00.095851+0000","last_clean":"2024-03-04T16:04:00.095851+0000","last_became_active":"2024-03-04T16:02:43.718570+0000","last_became_peered":"2024-03-04T16:02:43.718570+0000","last_unstale":"2024-03-04T16:04:00.095851+0000","last_undegraded":"2024-03-04T16:04:00.095851+0000","last_fullsized":"2024-03-04T16:04:00.095851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.354324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.354324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.540186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087619+0000","last_change":"2024-03-04T16:03:51.720621+0000","last_active":"2024-03-04T16:04:00.087619+0000","last_peered":"2024-03-04T16:04:00.087619+0000","last_clean":"2024-03-04T16:04:00.087619+0000","last_became_active":"2024-03-04T16:02:45.746864+0000","last_became_peered":"2024-03-04T16:02:45.746864+0000","last_unstale":"2024-03-04T16:04:00.087619+0000","last_undegraded":"2024-03-04T16:04:00.087619+0000","last_fullsized":"2024-03-04T16:04:00.087619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:51.720518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.720518+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.935701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171107+0000","last_change":"2024-03-04T16:03:57.893125+0000","last_active":"2024-03-04T16:04:00.171107+0000","last_peered":"2024-03-04T16:04:00.171107+0000","last_clean":"2024-03-04T16:04:00.171107+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:04:00.171107+0000","last_undegraded":"2024-03-04T16:04:00.171107+0000","last_fullsized":"2024-03-04T16:04:00.171107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.893007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.893007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.709209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.798890+0000","last_change":"2024-03-04T16:04:05.798912+0000","last_active":"2024-03-04T16:04:05.798890+0000","last_peered":"2024-03-04T16:04:05.798890+0000","last_clean":"2024-03-04T16:04:05.798890+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:04:05.798890+0000","last_undegraded":"2024-03-04T16:04:05.798890+0000","last_fullsized":"2024-03-04T16:04:05.798890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.798829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.798829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.585226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171475+0000","last_change":"2024-03-04T16:02:43.717093+0000","last_active":"2024-03-04T16:04:00.171475+0000","last_peered":"2024-03-04T16:04:00.171475+0000","last_clean":"2024-03-04T16:04:00.171475+0000","last_became_active":"2024-03-04T16:02:43.716542+0000","last_became_peered":"2024-03-04T16:02:43.716542+0000","last_unstale":"2024-03-04T16:04:00.171475+0000","last_undegraded":"2024-03-04T16:04:00.171475+0000","last_fullsized":"2024-03-04T16:04:00.171475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.808612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095860+0000","last_change":"2024-03-04T16:02:45.754181+0000","last_active":"2024-03-04T16:04:00.095860+0000","last_peered":"2024-03-04T16:04:00.095860+0000","last_clean":"2024-03-04T16:04:00.095860+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:04:00.095860+0000","last_undegraded":"2024-03-04T16:04:00.095860+0000","last_fullsized":"2024-03-04T16:04:00.095860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.606250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096015+0000","last_change":"2024-03-04T16:02:49.791755+0000","last_active":"2024-03-04T16:04:00.096015+0000","last_peered":"2024-03-04T16:04:00.096015+0000","last_clean":"2024-03-04T16:04:00.096015+0000","last_became_active":"2024-03-04T16:02:49.791216+0000","last_became_peered":"2024-03-04T16:02:49.791216+0000","last_unstale":"2024-03-04T16:04:00.096015+0000","last_undegraded":"2024-03-04T16:04:00.096015+0000","last_fullsized":"2024-03-04T16:04:00.096015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.261130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171216+0000","last_change":"2024-03-04T16:02:47.773727+0000","last_active":"2024-03-04T16:04:00.171216+0000","last_peered":"2024-03-04T16:04:00.171216+0000","last_clean":"2024-03-04T16:04:00.171216+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:04:00.171216+0000","last_undegraded":"2024-03-04T16:04:00.171216+0000","last_fullsized":"2024-03-04T16:04:00.171216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.258206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171414+0000","last_change":"2024-03-04T16:03:52.330168+0000","last_active":"2024-03-04T16:04:00.171414+0000","last_peered":"2024-03-04T16:04:00.171414+0000","last_clean":"2024-03-04T16:04:00.171414+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:04:00.171414+0000","last_undegraded":"2024-03-04T16:04:00.171414+0000","last_fullsized":"2024-03-04T16:04:00.171414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.330026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.330026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.392092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096129+0000","last_change":"2024-03-04T16:02:45.753666+0000","last_active":"2024-03-04T16:04:00.096129+0000","last_peered":"2024-03-04T16:04:00.096129+0000","last_clean":"2024-03-04T16:04:00.096129+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:04:00.096129+0000","last_undegraded":"2024-03-04T16:04:00.096129+0000","last_fullsized":"2024-03-04T16:04:00.096129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.790980+0000","last_change":"2024-03-04T16:04:03.791013+0000","last_active":"2024-03-04T16:04:03.790980+0000","last_peered":"2024-03-04T16:04:03.790980+0000","last_clean":"2024-03-04T16:04:03.790980+0000","last_became_active":"2024-03-04T16:02:49.794197+0000","last_became_peered":"2024-03-04T16:02:49.794197+0000","last_unstale":"2024-03-04T16:04:03.790980+0000","last_undegraded":"2024-03-04T16:04:03.790980+0000","last_fullsized":"2024-03-04T16:04:03.790980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.790905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.790905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.304942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096301+0000","last_change":"2024-03-04T16:02:47.777366+0000","last_active":"2024-03-04T16:04:00.096301+0000","last_peered":"2024-03-04T16:04:00.096301+0000","last_clean":"2024-03-04T16:04:00.096301+0000","last_became_active":"2024-03-04T16:02:47.777008+0000","last_became_peered":"2024-03-04T16:02:47.777008+0000","last_unstale":"2024-03-04T16:04:00.096301+0000","last_undegraded":"2024-03-04T16:04:00.096301+0000","last_fullsized":"2024-03-04T16:04:00.096301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.335199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172131+0000","last_change":"2024-03-04T16:02:43.711547+0000","last_active":"2024-03-04T16:04:00.172131+0000","last_peered":"2024-03-04T16:04:00.172131+0000","last_clean":"2024-03-04T16:04:00.172131+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:04:00.172131+0000","last_undegraded":"2024-03-04T16:04:00.172131+0000","last_fullsized":"2024-03-04T16:04:00.172131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.527444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.040200+0000","last_change":"2024-03-04T16:04:07.040237+0000","last_active":"2024-03-04T16:04:07.040200+0000","last_peered":"2024-03-04T16:04:07.040200+0000","last_clean":"2024-03-04T16:04:07.040200+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:04:07.040200+0000","last_undegraded":"2024-03-04T16:04:07.040200+0000","last_fullsized":"2024-03-04T16:04:07.040200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.040136+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.386047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095861+0000","last_change":"2024-03-04T16:02:49.792155+0000","last_active":"2024-03-04T16:04:00.095861+0000","last_peered":"2024-03-04T16:04:00.095861+0000","last_clean":"2024-03-04T16:04:00.095861+0000","last_became_active":"2024-03-04T16:02:49.791815+0000","last_became_peered":"2024-03-04T16:02:49.791815+0000","last_unstale":"2024-03-04T16:04:00.095861+0000","last_undegraded":"2024-03-04T16:04:00.095861+0000","last_fullsized":"2024-03-04T16:04:00.095861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.727768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170892+0000","last_change":"2024-03-04T16:03:51.939795+0000","last_active":"2024-03-04T16:04:00.170892+0000","last_peered":"2024-03-04T16:04:00.170892+0000","last_clean":"2024-03-04T16:04:00.170892+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:04:00.170892+0000","last_undegraded":"2024-03-04T16:04:00.170892+0000","last_fullsized":"2024-03-04T16:04:00.170892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.939712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.939712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.441874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171382+0000","last_change":"2024-03-04T16:03:44.419164+0000","last_active":"2024-03-04T16:04:00.171382+0000","last_peered":"2024-03-04T16:04:00.171382+0000","last_clean":"2024-03-04T16:04:00.171382+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:04:00.171382+0000","last_undegraded":"2024-03-04T16:04:00.171382+0000","last_fullsized":"2024-03-04T16:04:00.171382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:44.419065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:44.419065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.867222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171695+0000","last_change":"2024-03-04T16:03:50.400887+0000","last_active":"2024-03-04T16:04:00.171695+0000","last_peered":"2024-03-04T16:04:00.171695+0000","last_clean":"2024-03-04T16:04:00.171695+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:04:00.171695+0000","last_undegraded":"2024-03-04T16:04:00.171695+0000","last_fullsized":"2024-03-04T16:04:00.171695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:50.400784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.400784+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.749407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171573+0000","last_change":"2024-03-04T16:02:49.797503+0000","last_active":"2024-03-04T16:04:00.171573+0000","last_peered":"2024-03-04T16:04:00.171573+0000","last_clean":"2024-03-04T16:04:00.171573+0000","last_became_active":"2024-03-04T16:02:49.797174+0000","last_became_peered":"2024-03-04T16:02:49.797174+0000","last_unstale":"2024-03-04T16:04:00.171573+0000","last_undegraded":"2024-03-04T16:04:00.171573+0000","last_fullsized":"2024-03-04T16:04:00.171573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:17.229451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172538+0000","last_change":"2024-03-04T16:02:47.769130+0000","last_active":"2024-03-04T16:04:00.172538+0000","last_peered":"2024-03-04T16:04:00.172538+0000","last_clean":"2024-03-04T16:04:00.172538+0000","last_became_active":"2024-03-04T16:02:47.768874+0000","last_became_peered":"2024-03-04T16:02:47.768874+0000","last_unstale":"2024-03-04T16:04:00.172538+0000","last_undegraded":"2024-03-04T16:04:00.172538+0000","last_fullsized":"2024-03-04T16:04:00.172538+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.837345+0000","last_change":"2024-03-04T16:04:06.837374+0000","last_active":"2024-03-04T16:04:06.837345+0000","last_peered":"2024-03-04T16:04:06.837345+0000","last_clean":"2024-03-04T16:04:06.837345+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:04:06.837345+0000","last_undegraded":"2024-03-04T16:04:06.837345+0000","last_fullsized":"2024-03-04T16:04:06.837345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:06.837263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.837263+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.977815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096389+0000","last_change":"2024-03-04T16:02:45.749382+0000","last_active":"2024-03-04T16:04:00.096389+0000","last_peered":"2024-03-04T16:04:00.096389+0000","last_clean":"2024-03-04T16:04:00.096389+0000","last_became_active":"2024-03-04T16:02:45.748824+0000","last_became_peered":"2024-03-04T16:02:45.748824+0000","last_unstale":"2024-03-04T16:04:00.096389+0000","last_undegraded":"2024-03-04T16:04:00.096389+0000","last_fullsized":"2024-03-04T16:04:00.096389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.173149+0000","last_change":"2024-03-04T16:02:49.787819+0000","last_active":"2024-03-04T16:04:00.173149+0000","last_peered":"2024-03-04T16:04:00.173149+0000","last_clean":"2024-03-04T16:04:00.173149+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:04:00.173149+0000","last_undegraded":"2024-03-04T16:04:00.173149+0000","last_fullsized":"2024-03-04T16:04:00.173149+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.531354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096694+0000","last_change":"2024-03-04T16:02:47.774221+0000","last_active":"2024-03-04T16:04:00.096694+0000","last_peered":"2024-03-04T16:04:00.096694+0000","last_clean":"2024-03-04T16:04:00.096694+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:04:00.096694+0000","last_undegraded":"2024-03-04T16:04:00.096694+0000","last_fullsized":"2024-03-04T16:04:00.096694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.715311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170631+0000","last_change":"2024-03-04T16:03:49.810121+0000","last_active":"2024-03-04T16:04:00.170631+0000","last_peered":"2024-03-04T16:04:00.170631+0000","last_clean":"2024-03-04T16:04:00.170631+0000","last_became_active":"2024-03-04T16:02:43.713331+0000","last_became_peered":"2024-03-04T16:02:43.713331+0000","last_unstale":"2024-03-04T16:04:00.170631+0000","last_undegraded":"2024-03-04T16:04:00.170631+0000","last_fullsized":"2024-03-04T16:04:00.170631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:03:49.810002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:49.810002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.948952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172670+0000","last_change":"2024-03-04T16:03:55.306308+0000","last_active":"2024-03-04T16:04:00.172670+0000","last_peered":"2024-03-04T16:04:00.172670+0000","last_clean":"2024-03-04T16:04:00.172670+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:04:00.172670+0000","last_undegraded":"2024-03-04T16:04:00.172670+0000","last_fullsized":"2024-03-04T16:04:00.172670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:03:55.306180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.306180+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.830154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095503+0000","last_change":"2024-03-04T16:02:49.790101+0000","last_active":"2024-03-04T16:04:00.095503+0000","last_peered":"2024-03-04T16:04:00.095503+0000","last_clean":"2024-03-04T16:04:00.095503+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:04:00.095503+0000","last_undegraded":"2024-03-04T16:04:00.095503+0000","last_fullsized":"2024-03-04T16:04:00.095503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.704953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096828+0000","last_change":"2024-03-04T16:02:47.775318+0000","last_active":"2024-03-04T16:04:00.096828+0000","last_peered":"2024-03-04T16:04:00.096828+0000","last_clean":"2024-03-04T16:04:00.096828+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:04:00.096828+0000","last_undegraded":"2024-03-04T16:04:00.096828+0000","last_fullsized":"2024-03-04T16:04:00.096828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171544+0000","last_change":"2024-03-04T16:02:43.714055+0000","last_active":"2024-03-04T16:04:00.171544+0000","last_peered":"2024-03-04T16:04:00.171544+0000","last_clean":"2024-03-04T16:04:00.171544+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:04:00.171544+0000","last_undegraded":"2024-03-04T16:04:00.171544+0000","last_fullsized":"2024-03-04T16:04:00.171544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.115403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.755480+0000","last_change":"2024-03-04T16:04:05.755536+0000","last_active":"2024-03-04T16:04:05.755480+0000","last_peered":"2024-03-04T16:04:05.755480+0000","last_clean":"2024-03-04T16:04:05.755480+0000","last_became_active":"2024-03-04T16:02:45.745571+0000","last_became_peered":"2024-03-04T16:02:45.745571+0000","last_unstale":"2024-03-04T16:04:05.755480+0000","last_undegraded":"2024-03-04T16:04:05.755480+0000","last_fullsized":"2024-03-04T16:04:05.755480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:05.755370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.755370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.469566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172040+0000","last_change":"2024-03-04T16:02:49.787602+0000","last_active":"2024-03-04T16:04:00.172040+0000","last_peered":"2024-03-04T16:04:00.172040+0000","last_clean":"2024-03-04T16:04:00.172040+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:04:00.172040+0000","last_undegraded":"2024-03-04T16:04:00.172040+0000","last_fullsized":"2024-03-04T16:04:00.172040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.184909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171501+0000","last_change":"2024-03-04T16:02:47.770162+0000","last_active":"2024-03-04T16:04:00.171501+0000","last_peered":"2024-03-04T16:04:00.171501+0000","last_clean":"2024-03-04T16:04:00.171501+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:04:00.171501+0000","last_undegraded":"2024-03-04T16:04:00.171501+0000","last_fullsized":"2024-03-04T16:04:00.171501+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.513844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.086907+0000","last_change":"2024-03-04T16:03:46.718298+0000","last_active":"2024-03-04T16:04:00.086907+0000","last_peered":"2024-03-04T16:04:00.086907+0000","last_clean":"2024-03-04T16:04:00.086907+0000","last_became_active":"2024-03-04T16:02:43.718241+0000","last_became_peered":"2024-03-04T16:02:43.718241+0000","last_unstale":"2024-03-04T16:04:00.086907+0000","last_undegraded":"2024-03-04T16:04:00.086907+0000","last_fullsized":"2024-03-04T16:04:00.086907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.718188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.606455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096922+0000","last_change":"2024-03-04T16:03:52.421377+0000","last_active":"2024-03-04T16:04:00.096922+0000","last_peered":"2024-03-04T16:04:00.096922+0000","last_clean":"2024-03-04T16:04:00.096922+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:04:00.096922+0000","last_undegraded":"2024-03-04T16:04:00.096922+0000","last_fullsized":"2024-03-04T16:04:00.096922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.421285+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.703150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171188+0000","last_change":"2024-03-04T16:03:58.313071+0000","last_active":"2024-03-04T16:04:00.171188+0000","last_peered":"2024-03-04T16:04:00.171188+0000","last_clean":"2024-03-04T16:04:00.171188+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:04:00.171188+0000","last_undegraded":"2024-03-04T16:04:00.171188+0000","last_fullsized":"2024-03-04T16:04:00.171188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.312925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.312925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.584207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172545+0000","last_change":"2024-03-04T16:02:43.717223+0000","last_active":"2024-03-04T16:04:00.172545+0000","last_peered":"2024-03-04T16:04:00.172545+0000","last_clean":"2024-03-04T16:04:00.172545+0000","last_became_active":"2024-03-04T16:02:43.716945+0000","last_became_peered":"2024-03-04T16:02:43.716945+0000","last_unstale":"2024-03-04T16:04:00.172545+0000","last_undegraded":"2024-03-04T16:04:00.172545+0000","last_fullsized":"2024-03-04T16:04:00.172545+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.985563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170668+0000","last_change":"2024-03-04T16:03:48.260185+0000","last_active":"2024-03-04T16:04:00.170668+0000","last_peered":"2024-03-04T16:04:00.170668+0000","last_clean":"2024-03-04T16:04:00.170668+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:04:00.170668+0000","last_undegraded":"2024-03-04T16:04:00.170668+0000","last_fullsized":"2024-03-04T16:04:00.170668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:48.260047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.260047+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.815204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171856+0000","last_change":"2024-03-04T16:02:47.774009+0000","last_active":"2024-03-04T16:04:00.171856+0000","last_peered":"2024-03-04T16:04:00.171856+0000","last_clean":"2024-03-04T16:04:00.171856+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:04:00.171856+0000","last_undegraded":"2024-03-04T16:04:00.171856+0000","last_fullsized":"2024-03-04T16:04:00.171856+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096118+0000","last_change":"2024-03-04T16:02:49.796054+0000","last_active":"2024-03-04T16:04:00.096118+0000","last_peered":"2024-03-04T16:04:00.096118+0000","last_clean":"2024-03-04T16:04:00.096118+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:04:00.096118+0000","last_undegraded":"2024-03-04T16:04:00.096118+0000","last_fullsized":"2024-03-04T16:04:00.096118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.821795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172343+0000","last_change":"2024-03-04T16:03:55.817828+0000","last_active":"2024-03-04T16:04:00.172343+0000","last_peered":"2024-03-04T16:04:00.172343+0000","last_clean":"2024-03-04T16:04:00.172343+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:04:00.172343+0000","last_undegraded":"2024-03-04T16:04:00.172343+0000","last_fullsized":"2024-03-04T16:04:00.172343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:55.817709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.817709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.939046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171220+0000","last_change":"2024-03-04T16:03:48.407212+0000","last_active":"2024-03-04T16:04:00.171220+0000","last_peered":"2024-03-04T16:04:00.171220+0000","last_clean":"2024-03-04T16:04:00.171220+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:04:00.171220+0000","last_undegraded":"2024-03-04T16:04:00.171220+0000","last_fullsized":"2024-03-04T16:04:00.171220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:48.407088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.407088+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:49.238194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095123+0000","last_change":"2024-03-04T16:03:59.048702+0000","last_active":"2024-03-04T16:04:00.095123+0000","last_peered":"2024-03-04T16:04:00.095123+0000","last_clean":"2024-03-04T16:04:00.095123+0000","last_became_active":"2024-03-04T16:02:47.767316+0000","last_became_peered":"2024-03-04T16:02:47.767316+0000","last_unstale":"2024-03-04T16:04:00.095123+0000","last_undegraded":"2024-03-04T16:04:00.095123+0000","last_fullsized":"2024-03-04T16:04:00.095123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:04.396312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170696+0000","last_change":"2024-03-04T16:02:49.797192+0000","last_active":"2024-03-04T16:04:00.170696+0000","last_peered":"2024-03-04T16:04:00.170696+0000","last_clean":"2024-03-04T16:04:00.170696+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:04:00.170696+0000","last_undegraded":"2024-03-04T16:04:00.170696+0000","last_fullsized":"2024-03-04T16:04:00.170696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.448969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095320+0000","last_change":"2024-03-04T16:03:51.984808+0000","last_active":"2024-03-04T16:04:00.095320+0000","last_peered":"2024-03-04T16:04:00.095320+0000","last_clean":"2024-03-04T16:04:00.095320+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:04:00.095320+0000","last_undegraded":"2024-03-04T16:04:00.095320+0000","last_fullsized":"2024-03-04T16:04:00.095320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.984721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.984721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.092310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172295+0000","last_change":"2024-03-04T16:02:43.715496+0000","last_active":"2024-03-04T16:04:00.172295+0000","last_peered":"2024-03-04T16:04:00.172295+0000","last_clean":"2024-03-04T16:04:00.172295+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:04:00.172295+0000","last_undegraded":"2024-03-04T16:04:00.172295+0000","last_fullsized":"2024-03-04T16:04:00.172295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.926799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087201+0000","last_change":"2024-03-04T16:02:45.747239+0000","last_active":"2024-03-04T16:04:00.087201+0000","last_peered":"2024-03-04T16:04:00.087201+0000","last_clean":"2024-03-04T16:04:00.087201+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:04:00.087201+0000","last_undegraded":"2024-03-04T16:04:00.087201+0000","last_fullsized":"2024-03-04T16:04:00.087201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.287130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170394+0000","last_change":"2024-03-04T16:03:50.786884+0000","last_active":"2024-03-04T16:04:00.170394+0000","last_peered":"2024-03-04T16:04:00.170394+0000","last_clean":"2024-03-04T16:04:00.170394+0000","last_became_active":"2024-03-04T16:02:49.793023+0000","last_became_peered":"2024-03-04T16:02:49.793023+0000","last_unstale":"2024-03-04T16:04:00.170394+0000","last_undegraded":"2024-03-04T16:04:00.170394+0000","last_fullsized":"2024-03-04T16:04:00.170394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.786805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.786805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.570842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095255+0000","last_change":"2024-03-04T16:02:47.769558+0000","last_active":"2024-03-04T16:04:00.095255+0000","last_peered":"2024-03-04T16:04:00.095255+0000","last_clean":"2024-03-04T16:04:00.095255+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:04:00.095255+0000","last_undegraded":"2024-03-04T16:04:00.095255+0000","last_fullsized":"2024-03-04T16:04:00.095255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.736215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.230097+0000","last_change":"2024-03-04T16:04:06.230119+0000","last_active":"2024-03-04T16:04:06.230097+0000","last_peered":"2024-03-04T16:04:06.230097+0000","last_clean":"2024-03-04T16:04:06.230097+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:04:06.230097+0000","last_undegraded":"2024-03-04T16:04:06.230097+0000","last_fullsized":"2024-03-04T16:04:06.230097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.230030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.804736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171391+0000","last_change":"2024-03-04T16:02:45.752568+0000","last_active":"2024-03-04T16:04:00.171391+0000","last_peered":"2024-03-04T16:04:00.171391+0000","last_clean":"2024-03-04T16:04:00.171391+0000","last_became_active":"2024-03-04T16:02:45.752201+0000","last_became_peered":"2024-03-04T16:02:45.752201+0000","last_unstale":"2024-03-04T16:04:00.171391+0000","last_undegraded":"2024-03-04T16:04:00.171391+0000","last_fullsized":"2024-03-04T16:04:00.171391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.531681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096551+0000","last_change":"2024-03-04T16:02:49.793485+0000","last_active":"2024-03-04T16:04:00.096551+0000","last_peered":"2024-03-04T16:04:00.096551+0000","last_clean":"2024-03-04T16:04:00.096551+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:04:00.096551+0000","last_undegraded":"2024-03-04T16:04:00.096551+0000","last_fullsized":"2024-03-04T16:04:00.096551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.099839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170930+0000","last_change":"2024-03-04T16:03:54.344325+0000","last_active":"2024-03-04T16:04:00.170930+0000","last_peered":"2024-03-04T16:04:00.170930+0000","last_clean":"2024-03-04T16:04:00.170930+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:04:00.170930+0000","last_undegraded":"2024-03-04T16:04:00.170930+0000","last_fullsized":"2024-03-04T16:04:00.170930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:03:54.344193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:54.344193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.913538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171536+0000","last_change":"2024-03-04T16:03:59.437627+0000","last_active":"2024-03-04T16:04:00.171536+0000","last_peered":"2024-03-04T16:04:00.171536+0000","last_clean":"2024-03-04T16:04:00.171536+0000","last_became_active":"2024-03-04T16:02:43.717129+0000","last_became_peered":"2024-03-04T16:02:43.717129+0000","last_unstale":"2024-03-04T16:04:00.171536+0000","last_undegraded":"2024-03-04T16:04:00.171536+0000","last_fullsized":"2024-03-04T16:04:00.171536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.437524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.437524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.171391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":116,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171480+0000","last_change":"2024-03-04T16:02:45.753468+0000","last_active":"2024-03-04T16:04:00.171480+0000","last_peered":"2024-03-04T16:04:00.171480+0000","last_clean":"2024-03-04T16:04:00.171480+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:04:00.171480+0000","last_undegraded":"2024-03-04T16:04:00.171480+0000","last_fullsized":"2024-03-04T16:04:00.171480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.486508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171374+0000","last_change":"2024-03-04T16:02:47.773975+0000","last_active":"2024-03-04T16:04:00.171374+0000","last_peered":"2024-03-04T16:04:00.171374+0000","last_clean":"2024-03-04T16:04:00.171374+0000","last_became_active":"2024-03-04T16:02:47.773547+0000","last_became_peered":"2024-03-04T16:02:47.773547+0000","last_unstale":"2024-03-04T16:04:00.171374+0000","last_undegraded":"2024-03-04T16:04:00.171374+0000","last_fullsized":"2024-03-04T16:04:00.171374+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172366+0000","last_change":"2024-03-04T16:02:49.791079+0000","last_active":"2024-03-04T16:04:00.172366+0000","last_peered":"2024-03-04T16:04:00.172366+0000","last_clean":"2024-03-04T16:04:00.172366+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:04:00.172366+0000","last_undegraded":"2024-03-04T16:04:00.172366+0000","last_fullsized":"2024-03-04T16:04:00.172366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170995+0000","last_change":"2024-03-04T16:02:43.714877+0000","last_active":"2024-03-04T16:04:00.170995+0000","last_peered":"2024-03-04T16:04:00.170995+0000","last_clean":"2024-03-04T16:04:00.170995+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:04:00.170995+0000","last_undegraded":"2024-03-04T16:04:00.170995+0000","last_fullsized":"2024-03-04T16:04:00.170995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095150+0000","last_change":"2024-03-04T16:03:47.979184+0000","last_active":"2024-03-04T16:04:00.095150+0000","last_peered":"2024-03-04T16:04:00.095150+0000","last_clean":"2024-03-04T16:04:00.095150+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:04:00.095150+0000","last_undegraded":"2024-03-04T16:04:00.095150+0000","last_fullsized":"2024-03-04T16:04:00.095150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.979058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.979058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.315882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171982+0000","last_change":"2024-03-04T16:02:45.748924+0000","last_active":"2024-03-04T16:04:00.171982+0000","last_peered":"2024-03-04T16:04:00.171982+0000","last_clean":"2024-03-04T16:04:00.171982+0000","last_became_active":"2024-03-04T16:02:45.748615+0000","last_became_peered":"2024-03-04T16:02:45.748615+0000","last_unstale":"2024-03-04T16:04:00.171982+0000","last_undegraded":"2024-03-04T16:04:00.171982+0000","last_fullsized":"2024-03-04T16:04:00.171982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.429848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.048679+0000","last_change":"2024-03-04T16:04:03.048703+0000","last_active":"2024-03-04T16:04:03.048679+0000","last_peered":"2024-03-04T16:04:03.048679+0000","last_clean":"2024-03-04T16:04:03.048679+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:04:03.048679+0000","last_undegraded":"2024-03-04T16:04:03.048679+0000","last_fullsized":"2024-03-04T16:04:03.048679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.048620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.048620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.405283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172223+0000","last_change":"2024-03-04T16:02:49.794677+0000","last_active":"2024-03-04T16:04:00.172223+0000","last_peered":"2024-03-04T16:04:00.172223+0000","last_clean":"2024-03-04T16:04:00.172223+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:04:00.172223+0000","last_undegraded":"2024-03-04T16:04:00.172223+0000","last_fullsized":"2024-03-04T16:04:00.172223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.070330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170921+0000","last_change":"2024-03-04T16:03:12.851794+0000","last_active":"2024-03-04T16:04:00.170921+0000","last_peered":"2024-03-04T16:04:00.170921+0000","last_clean":"2024-03-04T16:04:00.170921+0000","last_became_active":"2024-03-04T16:02:34.079164+0000","last_became_peered":"2024-03-04T16:02:34.079164+0000","last_unstale":"2024-03-04T16:04:00.170921+0000","last_undegraded":"2024-03-04T16:04:00.170921+0000","last_fullsized":"2024-03-04T16:04:00.170921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'119","last_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:03:12.851677+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:36.356853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096256+0000","last_change":"2024-03-04T16:03:57.445763+0000","last_active":"2024-03-04T16:04:00.096256+0000","last_peered":"2024-03-04T16:04:00.096256+0000","last_clean":"2024-03-04T16:04:00.096256+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:04:00.096256+0000","last_undegraded":"2024-03-04T16:04:00.096256+0000","last_fullsized":"2024-03-04T16:04:00.096256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.445659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.445659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.402395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":98,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087582+0000","last_change":"2024-03-04T16:03:58.764928+0000","last_active":"2024-03-04T16:04:00.087582+0000","last_peered":"2024-03-04T16:04:00.087582+0000","last_clean":"2024-03-04T16:04:00.087582+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:04:00.087582+0000","last_undegraded":"2024-03-04T16:04:00.087582+0000","last_fullsized":"2024-03-04T16:04:00.087582+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:03:58.764821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.764821+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.352281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172055+0000","last_change":"2024-03-04T16:03:52.917188+0000","last_active":"2024-03-04T16:04:00.172055+0000","last_peered":"2024-03-04T16:04:00.172055+0000","last_clean":"2024-03-04T16:04:00.172055+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:04:00.172055+0000","last_undegraded":"2024-03-04T16:04:00.172055+0000","last_fullsized":"2024-03-04T16:04:00.172055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.917090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.917090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:00.018319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171270+0000","last_change":"2024-03-04T16:03:52.944487+0000","last_active":"2024-03-04T16:04:00.171270+0000","last_peered":"2024-03-04T16:04:00.171270+0000","last_clean":"2024-03-04T16:04:00.171270+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:04:00.171270+0000","last_undegraded":"2024-03-04T16:04:00.171270+0000","last_fullsized":"2024-03-04T16:04:00.171270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.944410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.132708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170566+0000","last_change":"2024-03-04T16:02:43.713670+0000","last_active":"2024-03-04T16:04:00.170566+0000","last_peered":"2024-03-04T16:04:00.170566+0000","last_clean":"2024-03-04T16:04:00.170566+0000","last_became_active":"2024-03-04T16:02:43.713276+0000","last_became_peered":"2024-03-04T16:02:43.713276+0000","last_unstale":"2024-03-04T16:04:00.170566+0000","last_undegraded":"2024-03-04T16:04:00.170566+0000","last_fullsized":"2024-03-04T16:04:00.170566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.212292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":145,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.136674+0000","last_change":"2024-03-04T16:02:46.749606+0000","last_active":"2024-03-04T16:04:06.136674+0000","last_peered":"2024-03-04T16:04:06.136674+0000","last_clean":"2024-03-04T16:04:06.136674+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:04:06.136674+0000","last_undegraded":"2024-03-04T16:04:06.136674+0000","last_fullsized":"2024-03-04T16:04:06.136674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:02:41.630915+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171663+0000","last_change":"2024-03-04T16:02:45.755785+0000","last_active":"2024-03-04T16:04:00.171663+0000","last_peered":"2024-03-04T16:04:00.171663+0000","last_clean":"2024-03-04T16:04:00.171663+0000","last_became_active":"2024-03-04T16:02:45.755435+0000","last_became_peered":"2024-03-04T16:02:45.755435+0000","last_unstale":"2024-03-04T16:04:00.171663+0000","last_undegraded":"2024-03-04T16:04:00.171663+0000","last_fullsized":"2024-03-04T16:04:00.171663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.974009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172637+0000","last_change":"2024-03-04T16:03:57.296517+0000","last_active":"2024-03-04T16:04:00.172637+0000","last_peered":"2024-03-04T16:04:00.172637+0000","last_clean":"2024-03-04T16:04:00.172637+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:04:00.172637+0000","last_undegraded":"2024-03-04T16:04:00.172637+0000","last_fullsized":"2024-03-04T16:04:00.172637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.296421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.296421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.272283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170706+0000","last_change":"2024-03-04T16:02:49.785263+0000","last_active":"2024-03-04T16:04:00.170706+0000","last_peered":"2024-03-04T16:04:00.170706+0000","last_clean":"2024-03-04T16:04:00.170706+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:04:00.170706+0000","last_undegraded":"2024-03-04T16:04:00.170706+0000","last_fullsized":"2024-03-04T16:04:00.170706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171392+0000","last_change":"2024-03-04T16:02:43.719367+0000","last_active":"2024-03-04T16:04:00.171392+0000","last_peered":"2024-03-04T16:04:00.171392+0000","last_clean":"2024-03-04T16:04:00.171392+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:04:00.171392+0000","last_undegraded":"2024-03-04T16:04:00.171392+0000","last_fullsized":"2024-03-04T16:04:00.171392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171241+0000","last_change":"2024-03-04T16:02:45.744516+0000","last_active":"2024-03-04T16:04:00.171241+0000","last_peered":"2024-03-04T16:04:00.171241+0000","last_clean":"2024-03-04T16:04:00.171241+0000","last_became_active":"2024-03-04T16:02:45.744172+0000","last_became_peered":"2024-03-04T16:02:45.744172+0000","last_unstale":"2024-03-04T16:04:00.171241+0000","last_undegraded":"2024-03-04T16:04:00.171241+0000","last_fullsized":"2024-03-04T16:04:00.171241+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.725177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087204+0000","last_change":"2024-03-04T16:03:50.770141+0000","last_active":"2024-03-04T16:04:00.087204+0000","last_peered":"2024-03-04T16:04:00.087204+0000","last_clean":"2024-03-04T16:04:00.087204+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:04:00.087204+0000","last_undegraded":"2024-03-04T16:04:00.087204+0000","last_fullsized":"2024-03-04T16:04:00.087204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.770048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.770048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.417482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170347+0000","last_change":"2024-03-04T16:02:49.797759+0000","last_active":"2024-03-04T16:04:00.170347+0000","last_peered":"2024-03-04T16:04:00.170347+0000","last_clean":"2024-03-04T16:04:00.170347+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:04:00.170347+0000","last_undegraded":"2024-03-04T16:04:00.170347+0000","last_fullsized":"2024-03-04T16:04:00.170347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.105311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172561+0000","last_change":"2024-03-04T16:02:43.711336+0000","last_active":"2024-03-04T16:04:00.172561+0000","last_peered":"2024-03-04T16:04:00.172561+0000","last_clean":"2024-03-04T16:04:00.172561+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:04:00.172561+0000","last_undegraded":"2024-03-04T16:04:00.172561+0000","last_fullsized":"2024-03-04T16:04:00.172561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:11.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170915+0000","last_change":"2024-03-04T16:02:45.748912+0000","last_active":"2024-03-04T16:04:00.170915+0000","last_peered":"2024-03-04T16:04:00.170915+0000","last_clean":"2024-03-04T16:04:00.170915+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:04:00.170915+0000","last_undegraded":"2024-03-04T16:04:00.170915+0000","last_fullsized":"2024-03-04T16:04:00.170915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171060+0000","last_change":"2024-03-04T16:02:47.777918+0000","last_active":"2024-03-04T16:04:00.171060+0000","last_peered":"2024-03-04T16:04:00.171060+0000","last_clean":"2024-03-04T16:04:00.171060+0000","last_became_active":"2024-03-04T16:02:47.777462+0000","last_became_peered":"2024-03-04T16:02:47.777462+0000","last_unstale":"2024-03-04T16:04:00.171060+0000","last_undegraded":"2024-03-04T16:04:00.171060+0000","last_fullsized":"2024-03-04T16:04:00.171060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.300423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171829+0000","last_change":"2024-03-04T16:03:56.858487+0000","last_active":"2024-03-04T16:04:00.171829+0000","last_peered":"2024-03-04T16:04:00.171829+0000","last_clean":"2024-03-04T16:04:00.171829+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:04:00.171829+0000","last_undegraded":"2024-03-04T16:04:00.171829+0000","last_fullsized":"2024-03-04T16:04:00.171829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.858376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.858376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.769864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171668+0000","last_change":"2024-03-04T16:02:43.712924+0000","last_active":"2024-03-04T16:04:00.171668+0000","last_peered":"2024-03-04T16:04:00.171668+0000","last_clean":"2024-03-04T16:04:00.171668+0000","last_became_active":"2024-03-04T16:02:43.712287+0000","last_became_peered":"2024-03-04T16:02:43.712287+0000","last_unstale":"2024-03-04T16:04:00.171668+0000","last_undegraded":"2024-03-04T16:04:00.171668+0000","last_fullsized":"2024-03-04T16:04:00.171668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.101366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.515895+0000","last_change":"2024-03-04T16:04:06.515940+0000","last_active":"2024-03-04T16:04:06.515895+0000","last_peered":"2024-03-04T16:04:06.515895+0000","last_clean":"2024-03-04T16:04:06.515895+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:04:06.515895+0000","last_undegraded":"2024-03-04T16:04:06.515895+0000","last_fullsized":"2024-03-04T16:04:06.515895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.515823+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.602986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096127+0000","last_change":"2024-03-04T16:02:47.767792+0000","last_active":"2024-03-04T16:04:00.096127+0000","last_peered":"2024-03-04T16:04:00.096127+0000","last_clean":"2024-03-04T16:04:00.096127+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:04:00.096127+0000","last_undegraded":"2024-03-04T16:04:00.096127+0000","last_fullsized":"2024-03-04T16:04:00.096127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171503+0000","last_change":"2024-03-04T16:02:49.789412+0000","last_active":"2024-03-04T16:04:00.171503+0000","last_peered":"2024-03-04T16:04:00.171503+0000","last_clean":"2024-03-04T16:04:00.171503+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:04:00.171503+0000","last_undegraded":"2024-03-04T16:04:00.171503+0000","last_fullsized":"2024-03-04T16:04:00.171503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.799791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096671+0000","last_change":"2024-03-04T16:02:43.716171+0000","last_active":"2024-03-04T16:04:00.096671+0000","last_peered":"2024-03-04T16:04:00.096671+0000","last_clean":"2024-03-04T16:04:00.096671+0000","last_became_active":"2024-03-04T16:02:43.715346+0000","last_became_peered":"2024-03-04T16:02:43.715346+0000","last_unstale":"2024-03-04T16:04:00.096671+0000","last_undegraded":"2024-03-04T16:04:00.096671+0000","last_fullsized":"2024-03-04T16:04:00.096671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095990+0000","last_change":"2024-03-04T16:02:45.749977+0000","last_active":"2024-03-04T16:04:00.095990+0000","last_peered":"2024-03-04T16:04:00.095990+0000","last_clean":"2024-03-04T16:04:00.095990+0000","last_became_active":"2024-03-04T16:02:45.749633+0000","last_became_peered":"2024-03-04T16:02:45.749633+0000","last_unstale":"2024-03-04T16:04:00.095990+0000","last_undegraded":"2024-03-04T16:04:00.095990+0000","last_fullsized":"2024-03-04T16:04:00.095990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.646959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170756+0000","last_change":"2024-03-04T16:02:47.768154+0000","last_active":"2024-03-04T16:04:00.170756+0000","last_peered":"2024-03-04T16:04:00.170756+0000","last_clean":"2024-03-04T16:04:00.170756+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:04:00.170756+0000","last_undegraded":"2024-03-04T16:04:00.170756+0000","last_fullsized":"2024-03-04T16:04:00.170756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.581920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172282+0000","last_change":"2024-03-04T16:02:49.792150+0000","last_active":"2024-03-04T16:04:00.172282+0000","last_peered":"2024-03-04T16:04:00.172282+0000","last_clean":"2024-03-04T16:04:00.172282+0000","last_became_active":"2024-03-04T16:02:49.791851+0000","last_became_peered":"2024-03-04T16:02:49.791851+0000","last_unstale":"2024-03-04T16:04:00.172282+0000","last_undegraded":"2024-03-04T16:04:00.172282+0000","last_fullsized":"2024-03-04T16:04:00.172282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.262236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.173081+0000","last_change":"2024-03-04T16:02:47.768859+0000","last_active":"2024-03-04T16:04:00.173081+0000","last_peered":"2024-03-04T16:04:00.173081+0000","last_clean":"2024-03-04T16:04:00.173081+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:04:00.173081+0000","last_undegraded":"2024-03-04T16:04:00.173081+0000","last_fullsized":"2024-03-04T16:04:00.173081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.459494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095980+0000","last_change":"2024-03-04T16:03:58.462106+0000","last_active":"2024-03-04T16:04:00.095980+0000","last_peered":"2024-03-04T16:04:00.095980+0000","last_clean":"2024-03-04T16:04:00.095980+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:04:00.095980+0000","last_undegraded":"2024-03-04T16:04:00.095980+0000","last_fullsized":"2024-03-04T16:04:00.095980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.462018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:08.477439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171988+0000","last_change":"2024-03-04T16:03:46.963168+0000","last_active":"2024-03-04T16:04:00.171988+0000","last_peered":"2024-03-04T16:04:00.171988+0000","last_clean":"2024-03-04T16:04:00.171988+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:04:00.171988+0000","last_undegraded":"2024-03-04T16:04:00.171988+0000","last_fullsized":"2024-03-04T16:04:00.171988+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:46.963065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.963065+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.801631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096277+0000","last_change":"2024-03-04T16:02:49.794003+0000","last_active":"2024-03-04T16:04:00.096277+0000","last_peered":"2024-03-04T16:04:00.096277+0000","last_clean":"2024-03-04T16:04:00.096277+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:04:00.096277+0000","last_undegraded":"2024-03-04T16:04:00.096277+0000","last_fullsized":"2024-03-04T16:04:00.096277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.636557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172548+0000","last_change":"2024-03-04T16:02:47.766009+0000","last_active":"2024-03-04T16:04:00.172548+0000","last_peered":"2024-03-04T16:04:00.172548+0000","last_clean":"2024-03-04T16:04:00.172548+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:04:00.172548+0000","last_undegraded":"2024-03-04T16:04:00.172548+0000","last_fullsized":"2024-03-04T16:04:00.172548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.260597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170870+0000","last_change":"2024-03-04T16:02:43.705402+0000","last_active":"2024-03-04T16:04:00.170870+0000","last_peered":"2024-03-04T16:04:00.170870+0000","last_clean":"2024-03-04T16:04:00.170870+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:04:00.170870+0000","last_undegraded":"2024-03-04T16:04:00.170870+0000","last_fullsized":"2024-03-04T16:04:00.170870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.029120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172915+0000","last_change":"2024-03-04T16:02:45.746393+0000","last_active":"2024-03-04T16:04:00.172915+0000","last_peered":"2024-03-04T16:04:00.172915+0000","last_clean":"2024-03-04T16:04:00.172915+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:04:00.172915+0000","last_undegraded":"2024-03-04T16:04:00.172915+0000","last_fullsized":"2024-03-04T16:04:00.172915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.554587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.063138+0000","last_change":"2024-03-04T16:04:07.063161+0000","last_active":"2024-03-04T16:04:07.063138+0000","last_peered":"2024-03-04T16:04:07.063138+0000","last_clean":"2024-03-04T16:04:07.063138+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:04:07.063138+0000","last_undegraded":"2024-03-04T16:04:07.063138+0000","last_fullsized":"2024-03-04T16:04:07.063138+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.063063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.063063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.152553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.976020+0000","last_change":"2024-03-04T16:04:04.976057+0000","last_active":"2024-03-04T16:04:04.976020+0000","last_peered":"2024-03-04T16:04:04.976020+0000","last_clean":"2024-03-04T16:04:04.976020+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:04:04.976020+0000","last_undegraded":"2024-03-04T16:04:04.976020+0000","last_fullsized":"2024-03-04T16:04:04.976020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.975944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.975944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.116230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087321+0000","last_change":"2024-03-04T16:02:43.716227+0000","last_active":"2024-03-04T16:04:00.087321+0000","last_peered":"2024-03-04T16:04:00.087321+0000","last_clean":"2024-03-04T16:04:00.087321+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:04:00.087321+0000","last_undegraded":"2024-03-04T16:04:00.087321+0000","last_fullsized":"2024-03-04T16:04:00.087321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.851707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172875+0000","last_change":"2024-03-04T16:02:45.743746+0000","last_active":"2024-03-04T16:04:00.172875+0000","last_peered":"2024-03-04T16:04:00.172875+0000","last_clean":"2024-03-04T16:04:00.172875+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:04:00.172875+0000","last_undegraded":"2024-03-04T16:04:00.172875+0000","last_fullsized":"2024-03-04T16:04:00.172875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.769608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.509705+0000","last_change":"2024-03-04T16:04:04.509748+0000","last_active":"2024-03-04T16:04:04.509705+0000","last_peered":"2024-03-04T16:04:04.509705+0000","last_clean":"2024-03-04T16:04:04.509705+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:04:04.509705+0000","last_undegraded":"2024-03-04T16:04:04.509705+0000","last_fullsized":"2024-03-04T16:04:04.509705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.509608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.509608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.020538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095459+0000","last_change":"2024-03-04T16:02:47.770597+0000","last_active":"2024-03-04T16:04:00.095459+0000","last_peered":"2024-03-04T16:04:00.095459+0000","last_clean":"2024-03-04T16:04:00.095459+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:04:00.095459+0000","last_undegraded":"2024-03-04T16:04:00.095459+0000","last_fullsized":"2024-03-04T16:04:00.095459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.503152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096393+0000","last_change":"2024-03-04T16:02:43.715105+0000","last_active":"2024-03-04T16:04:00.096393+0000","last_peered":"2024-03-04T16:04:00.096393+0000","last_clean":"2024-03-04T16:04:00.096393+0000","last_became_active":"2024-03-04T16:02:43.714792+0000","last_became_peered":"2024-03-04T16:02:43.714792+0000","last_unstale":"2024-03-04T16:04:00.096393+0000","last_undegraded":"2024-03-04T16:04:00.096393+0000","last_fullsized":"2024-03-04T16:04:00.096393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171318+0000","last_change":"2024-03-04T16:02:45.744111+0000","last_active":"2024-03-04T16:04:00.171318+0000","last_peered":"2024-03-04T16:04:00.171318+0000","last_clean":"2024-03-04T16:04:00.171318+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:04:00.171318+0000","last_undegraded":"2024-03-04T16:04:00.171318+0000","last_fullsized":"2024-03-04T16:04:00.171318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.765686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070485,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":946996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1388544,"data_stored":944684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":856064,"data_stored":356088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1413120,"data_stored":946340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":790528,"data_stored":354816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":356082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":355306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626463,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":354642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:04:08.474 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T16:04:08.475 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-04T16:04:08.475 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-04T16:04:08.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-04T16:04:08.724 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:04:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:04:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 4.12 deep-scrub starts 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 4.12 deep-scrub ok 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[29779]: from='client.14769 v1:172.21.15.38:0/2846190584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 4.12 deep-scrub starts 2024-03-04T16:04:08.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 4.12 deep-scrub ok 2024-03-04T16:04:08.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:04:08.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:04:08.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:08.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:08 smithi038 ceph-mon[38113]: from='client.14769 v1:172.21.15.38:0/2846190584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:04:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:04:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 4.12 deep-scrub starts 2024-03-04T16:04:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 4.12 deep-scrub ok 2024-03-04T16:04:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:04:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:04:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:08 smithi176 ceph-mon[33619]: from='client.14769 v1:172.21.15.38:0/2846190584' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 3.a deep-scrub starts 2024-03-04T16:04:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 3.a deep-scrub ok 2024-03-04T16:04:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:04:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:04:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:09 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:04:09.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 3.a deep-scrub starts 2024-03-04T16:04:09.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 3.a deep-scrub ok 2024-03-04T16:04:09.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:04:09.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 3.a deep-scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 3.a deep-scrub ok 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:04:09.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:04:09.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:09 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:04:10.678 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:10.678 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-04T16:04:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:04:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:04:10.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:04:10.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:10 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:04:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:04:10.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:04:10.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:10 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:04:11.658 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-04T16:04:11.658 INFO:tasks.cephadm:Setup complete, yielding 2024-03-04T16:04:11.659 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-04T16:04:11.670 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-04T16:04:11.670 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-asm0deuz-testing-2024-03-04-1211'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-asm0deuz-testing-2024-03-04-1211', 'sha1': '9ca0b5de35ed230e84387abdd8e619ed8e4f3282'}} 2024-03-04T16:04:11.670 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-04T16:04:11.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-04T16:04:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:04:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:04:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:04:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:04:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:04:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:04:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1380850486' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:04:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:04:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:11 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:04:11.924 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:04:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:04:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:04:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:04:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1380850486' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:04:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:04:11.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1380850486' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-04T16:04:11.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:04:11.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:11 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:04:12.598 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:12 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:04:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:12 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:04:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:12 smithi176 ceph-mon[33619]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-04T16:04:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:04:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:04:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[29779]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-04T16:04:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:04:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:04:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:12 smithi038 ceph-mon[38113]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-03-04T16:04:13.605 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:13 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:04:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:13 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:04:13.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:13.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:04:13.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:04:13.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE Bus STOPPING 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE Bus STOPPED 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE Bus STARTING 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE Serving on http://:::9283 2024-03-04T16:04:13.884 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:04:13] ENGINE Bus STARTED 2024-03-04T16:04:14.344 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-04T16:04:14.345 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-04T16:04:14.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:04:14.591 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:14.610 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:04:14.432Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003314934s 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:04:14.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[29779]: from='client.34658 ' entity='client.admin' 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:04:14.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:14 smithi038 ceph-mon[38113]: from='client.34658 ' entity='client.admin' 2024-03-04T16:04:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:04:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:04:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:14 smithi176 ceph-mon[33619]: from='client.34658 ' entity='client.admin' 2024-03-04T16:04:15.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:04:15.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:04:15.882 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:15 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:04:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:04:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:04:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:04:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:04:16.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:04:16.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:15 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:04:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:16.426 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:04:16.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:04:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:04:16.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:16 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:04:17.043 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-04T16:04:15.293274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1017,"num_read_kb":863,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240892,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720324,"statfs":{"total":767960285184,"available":767713611776,"internally_reserved":0,"allocated":8318976,"data_stored":4614954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002964"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171404+0000","last_change":"2024-03-04T16:03:57.477127+0000","last_active":"2024-03-04T16:04:00.171404+0000","last_peered":"2024-03-04T16:04:00.171404+0000","last_clean":"2024-03-04T16:04:00.171404+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:04:00.171404+0000","last_undegraded":"2024-03-04T16:04:00.171404+0000","last_fullsized":"2024-03-04T16:04:00.171404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.477014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.477014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.118199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170834+0000","last_change":"2024-03-04T16:02:47.767863+0000","last_active":"2024-03-04T16:04:00.170834+0000","last_peered":"2024-03-04T16:04:00.170834+0000","last_clean":"2024-03-04T16:04:00.170834+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:04:00.170834+0000","last_undegraded":"2024-03-04T16:04:00.170834+0000","last_fullsized":"2024-03-04T16:04:00.170834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.395983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171558+0000","last_change":"2024-03-04T16:03:45.773330+0000","last_active":"2024-03-04T16:04:00.171558+0000","last_peered":"2024-03-04T16:04:00.171558+0000","last_clean":"2024-03-04T16:04:00.171558+0000","last_became_active":"2024-03-04T16:02:43.714846+0000","last_became_peered":"2024-03-04T16:02:43.714846+0000","last_unstale":"2024-03-04T16:04:00.171558+0000","last_undegraded":"2024-03-04T16:04:00.171558+0000","last_fullsized":"2024-03-04T16:04:00.171558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:45.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.773213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.488104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170701+0000","last_change":"2024-03-04T16:02:45.748421+0000","last_active":"2024-03-04T16:04:00.170701+0000","last_peered":"2024-03-04T16:04:00.170701+0000","last_clean":"2024-03-04T16:04:00.170701+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:04:00.170701+0000","last_undegraded":"2024-03-04T16:04:00.170701+0000","last_fullsized":"2024-03-04T16:04:00.170701+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.620264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.380440+0000","last_change":"2024-03-04T16:04:05.380482+0000","last_active":"2024-03-04T16:04:05.380440+0000","last_peered":"2024-03-04T16:04:05.380440+0000","last_clean":"2024-03-04T16:04:05.380440+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:04:05.380440+0000","last_undegraded":"2024-03-04T16:04:05.380440+0000","last_fullsized":"2024-03-04T16:04:05.380440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.380364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.380364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.798710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.519799+0000","last_change":"2024-03-04T16:04:08.519836+0000","last_active":"2024-03-04T16:04:08.519799+0000","last_peered":"2024-03-04T16:04:08.519799+0000","last_clean":"2024-03-04T16:04:08.519799+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:04:08.519799+0000","last_undegraded":"2024-03-04T16:04:08.519799+0000","last_fullsized":"2024-03-04T16:04:08.519799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.519713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.519713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.148055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096811+0000","last_change":"2024-03-04T16:02:43.712774+0000","last_active":"2024-03-04T16:04:00.096811+0000","last_peered":"2024-03-04T16:04:00.096811+0000","last_clean":"2024-03-04T16:04:00.096811+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:04:00.096811+0000","last_undegraded":"2024-03-04T16:04:00.096811+0000","last_fullsized":"2024-03-04T16:04:00.096811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.121882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096197+0000","last_change":"2024-03-04T16:03:50.379226+0000","last_active":"2024-03-04T16:04:00.096197+0000","last_peered":"2024-03-04T16:04:00.096197+0000","last_clean":"2024-03-04T16:04:00.096197+0000","last_became_active":"2024-03-04T16:02:45.749383+0000","last_became_peered":"2024-03-04T16:02:45.749383+0000","last_unstale":"2024-03-04T16:04:00.096197+0000","last_undegraded":"2024-03-04T16:04:00.096197+0000","last_fullsized":"2024-03-04T16:04:00.096197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:50.379077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.379077+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.153393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170489+0000","last_change":"2024-03-04T16:03:56.851189+0000","last_active":"2024-03-04T16:04:00.170489+0000","last_peered":"2024-03-04T16:04:00.170489+0000","last_clean":"2024-03-04T16:04:00.170489+0000","last_became_active":"2024-03-04T16:02:49.791142+0000","last_became_peered":"2024-03-04T16:02:49.791142+0000","last_unstale":"2024-03-04T16:04:00.170489+0000","last_undegraded":"2024-03-04T16:04:00.170489+0000","last_fullsized":"2024-03-04T16:04:00.170489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.851094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.851094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.117129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171309+0000","last_change":"2024-03-04T16:02:47.775367+0000","last_active":"2024-03-04T16:04:00.171309+0000","last_peered":"2024-03-04T16:04:00.171309+0000","last_clean":"2024-03-04T16:04:00.171309+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:04:00.171309+0000","last_undegraded":"2024-03-04T16:04:00.171309+0000","last_fullsized":"2024-03-04T16:04:00.171309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.508809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172396+0000","last_change":"2024-03-04T16:03:51.290213+0000","last_active":"2024-03-04T16:04:00.172396+0000","last_peered":"2024-03-04T16:04:00.172396+0000","last_clean":"2024-03-04T16:04:00.172396+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:04:00.172396+0000","last_undegraded":"2024-03-04T16:04:00.172396+0000","last_fullsized":"2024-03-04T16:04:00.172396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.290117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.290117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.885059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.434104+0000","last_change":"2024-03-04T16:04:00.434148+0000","last_active":"2024-03-04T16:04:00.434104+0000","last_peered":"2024-03-04T16:04:00.434104+0000","last_clean":"2024-03-04T16:04:00.434104+0000","last_became_active":"2024-03-04T16:02:45.745532+0000","last_became_peered":"2024-03-04T16:02:45.745532+0000","last_unstale":"2024-03-04T16:04:00.434104+0000","last_undegraded":"2024-03-04T16:04:00.434104+0000","last_fullsized":"2024-03-04T16:04:00.434104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.434035+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.035919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:02.311200+0000","last_change":"2024-03-04T16:04:02.311228+0000","last_active":"2024-03-04T16:04:02.311200+0000","last_peered":"2024-03-04T16:04:02.311200+0000","last_clean":"2024-03-04T16:04:02.311200+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:04:02.311200+0000","last_undegraded":"2024-03-04T16:04:02.311200+0000","last_fullsized":"2024-03-04T16:04:02.311200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:02.311103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:02.311103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:06.585200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.905847+0000","last_change":"2024-03-04T16:04:08.905885+0000","last_active":"2024-03-04T16:04:08.905847+0000","last_peered":"2024-03-04T16:04:08.905847+0000","last_clean":"2024-03-04T16:04:08.905847+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:04:08.905847+0000","last_undegraded":"2024-03-04T16:04:08.905847+0000","last_fullsized":"2024-03-04T16:04:08.905847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.905768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.905768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.441912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.292580+0000","last_change":"2024-03-04T16:04:00.292627+0000","last_active":"2024-03-04T16:04:00.292580+0000","last_peered":"2024-03-04T16:04:00.292580+0000","last_clean":"2024-03-04T16:04:00.292580+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:04:00.292580+0000","last_undegraded":"2024-03-04T16:04:00.292580+0000","last_fullsized":"2024-03-04T16:04:00.292580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:00.292499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.292499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.161517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171851+0000","last_change":"2024-03-04T16:02:45.748750+0000","last_active":"2024-03-04T16:04:00.171851+0000","last_peered":"2024-03-04T16:04:00.171851+0000","last_clean":"2024-03-04T16:04:00.171851+0000","last_became_active":"2024-03-04T16:02:45.747787+0000","last_became_peered":"2024-03-04T16:02:45.747787+0000","last_unstale":"2024-03-04T16:04:00.171851+0000","last_undegraded":"2024-03-04T16:04:00.171851+0000","last_fullsized":"2024-03-04T16:04:00.171851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:14.663457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172933+0000","last_change":"2024-03-04T16:02:49.785706+0000","last_active":"2024-03-04T16:04:00.172933+0000","last_peered":"2024-03-04T16:04:00.172933+0000","last_clean":"2024-03-04T16:04:00.172933+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:04:00.172933+0000","last_undegraded":"2024-03-04T16:04:00.172933+0000","last_fullsized":"2024-03-04T16:04:00.172933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.708627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.039716+0000","last_change":"2024-03-04T16:04:08.039737+0000","last_active":"2024-03-04T16:04:08.039716+0000","last_peered":"2024-03-04T16:04:08.039716+0000","last_clean":"2024-03-04T16:04:08.039716+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:04:08.039716+0000","last_undegraded":"2024-03-04T16:04:08.039716+0000","last_fullsized":"2024-03-04T16:04:08.039716+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.039649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.039649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.929286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171158+0000","last_change":"2024-03-04T16:02:43.714700+0000","last_active":"2024-03-04T16:04:00.171158+0000","last_peered":"2024-03-04T16:04:00.171158+0000","last_clean":"2024-03-04T16:04:00.171158+0000","last_became_active":"2024-03-04T16:02:43.714419+0000","last_became_peered":"2024-03-04T16:02:43.714419+0000","last_unstale":"2024-03-04T16:04:00.171158+0000","last_undegraded":"2024-03-04T16:04:00.171158+0000","last_fullsized":"2024-03-04T16:04:00.171158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.557877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095293+0000","last_change":"2024-03-04T16:03:45.980725+0000","last_active":"2024-03-04T16:04:00.095293+0000","last_peered":"2024-03-04T16:04:00.095293+0000","last_clean":"2024-03-04T16:04:00.095293+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:04:00.095293+0000","last_undegraded":"2024-03-04T16:04:00.095293+0000","last_fullsized":"2024-03-04T16:04:00.095293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:45.980606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.980606+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:01.702710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096882+0000","last_change":"2024-03-04T16:02:49.794035+0000","last_active":"2024-03-04T16:04:00.096882+0000","last_peered":"2024-03-04T16:04:00.096882+0000","last_clean":"2024-03-04T16:04:00.096882+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:04:00.096882+0000","last_undegraded":"2024-03-04T16:04:00.096882+0000","last_fullsized":"2024-03-04T16:04:00.096882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.379614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171229+0000","last_change":"2024-03-04T16:02:47.769883+0000","last_active":"2024-03-04T16:04:00.171229+0000","last_peered":"2024-03-04T16:04:00.171229+0000","last_clean":"2024-03-04T16:04:00.171229+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:04:00.171229+0000","last_undegraded":"2024-03-04T16:04:00.171229+0000","last_fullsized":"2024-03-04T16:04:00.171229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.833777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171760+0000","last_change":"2024-03-04T16:03:46.423866+0000","last_active":"2024-03-04T16:04:00.171760+0000","last_peered":"2024-03-04T16:04:00.171760+0000","last_clean":"2024-03-04T16:04:00.171760+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:04:00.171760+0000","last_undegraded":"2024-03-04T16:04:00.171760+0000","last_fullsized":"2024-03-04T16:04:00.171760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.423767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.868075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096446+0000","last_change":"2024-03-04T16:03:55.411658+0000","last_active":"2024-03-04T16:04:00.096446+0000","last_peered":"2024-03-04T16:04:00.096446+0000","last_clean":"2024-03-04T16:04:00.096446+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:04:00.096446+0000","last_undegraded":"2024-03-04T16:04:00.096446+0000","last_fullsized":"2024-03-04T16:04:00.096446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:55.411554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.411554+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.742258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171968+0000","last_change":"2024-03-04T16:02:49.792726+0000","last_active":"2024-03-04T16:04:00.171968+0000","last_peered":"2024-03-04T16:04:00.171968+0000","last_clean":"2024-03-04T16:04:00.171968+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:04:00.171968+0000","last_undegraded":"2024-03-04T16:04:00.171968+0000","last_fullsized":"2024-03-04T16:04:00.171968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.657727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.707325+0000","last_change":"2024-03-04T16:04:09.707358+0000","last_active":"2024-03-04T16:04:09.707325+0000","last_peered":"2024-03-04T16:04:09.707325+0000","last_clean":"2024-03-04T16:04:09.707325+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:04:09.707325+0000","last_undegraded":"2024-03-04T16:04:09.707325+0000","last_fullsized":"2024-03-04T16:04:09.707325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.707234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.707234+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.027920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:01.014974+0000","last_change":"2024-03-04T16:04:01.015009+0000","last_active":"2024-03-04T16:04:01.014974+0000","last_peered":"2024-03-04T16:04:01.014974+0000","last_clean":"2024-03-04T16:04:01.014974+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:04:01.014974+0000","last_undegraded":"2024-03-04T16:04:01.014974+0000","last_fullsized":"2024-03-04T16:04:01.014974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:01.014871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:01.014871+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:13.772804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171781+0000","last_change":"2024-03-04T16:02:45.745063+0000","last_active":"2024-03-04T16:04:00.171781+0000","last_peered":"2024-03-04T16:04:00.171781+0000","last_clean":"2024-03-04T16:04:00.171781+0000","last_became_active":"2024-03-04T16:02:45.744705+0000","last_became_peered":"2024-03-04T16:02:45.744705+0000","last_unstale":"2024-03-04T16:04:00.171781+0000","last_undegraded":"2024-03-04T16:04:00.171781+0000","last_fullsized":"2024-03-04T16:04:00.171781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.665716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171960+0000","last_change":"2024-03-04T16:03:53.955279+0000","last_active":"2024-03-04T16:04:00.171960+0000","last_peered":"2024-03-04T16:04:00.171960+0000","last_clean":"2024-03-04T16:04:00.171960+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:04:00.171960+0000","last_undegraded":"2024-03-04T16:04:00.171960+0000","last_fullsized":"2024-03-04T16:04:00.171960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:53.955186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:53.955186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.764558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171905+0000","last_change":"2024-03-04T16:03:57.874266+0000","last_active":"2024-03-04T16:04:00.171905+0000","last_peered":"2024-03-04T16:04:00.171905+0000","last_clean":"2024-03-04T16:04:00.171905+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:04:00.171905+0000","last_undegraded":"2024-03-04T16:04:00.171905+0000","last_fullsized":"2024-03-04T16:04:00.171905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.874150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.874150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.705755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095851+0000","last_change":"2024-03-04T16:03:47.354437+0000","last_active":"2024-03-04T16:04:00.095851+0000","last_peered":"2024-03-04T16:04:00.095851+0000","last_clean":"2024-03-04T16:04:00.095851+0000","last_became_active":"2024-03-04T16:02:43.718570+0000","last_became_peered":"2024-03-04T16:02:43.718570+0000","last_unstale":"2024-03-04T16:04:00.095851+0000","last_undegraded":"2024-03-04T16:04:00.095851+0000","last_fullsized":"2024-03-04T16:04:00.095851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.354324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.354324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.540186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087619+0000","last_change":"2024-03-04T16:03:51.720621+0000","last_active":"2024-03-04T16:04:00.087619+0000","last_peered":"2024-03-04T16:04:00.087619+0000","last_clean":"2024-03-04T16:04:00.087619+0000","last_became_active":"2024-03-04T16:02:45.746864+0000","last_became_peered":"2024-03-04T16:02:45.746864+0000","last_unstale":"2024-03-04T16:04:00.087619+0000","last_undegraded":"2024-03-04T16:04:00.087619+0000","last_fullsized":"2024-03-04T16:04:00.087619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:51.720518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.720518+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.935701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171107+0000","last_change":"2024-03-04T16:03:57.893125+0000","last_active":"2024-03-04T16:04:00.171107+0000","last_peered":"2024-03-04T16:04:00.171107+0000","last_clean":"2024-03-04T16:04:00.171107+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:04:00.171107+0000","last_undegraded":"2024-03-04T16:04:00.171107+0000","last_fullsized":"2024-03-04T16:04:00.171107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.893007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.893007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.709209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.798890+0000","last_change":"2024-03-04T16:04:05.798912+0000","last_active":"2024-03-04T16:04:05.798890+0000","last_peered":"2024-03-04T16:04:05.798890+0000","last_clean":"2024-03-04T16:04:05.798890+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:04:05.798890+0000","last_undegraded":"2024-03-04T16:04:05.798890+0000","last_fullsized":"2024-03-04T16:04:05.798890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.798829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.798829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.585226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.761398+0000","last_change":"2024-03-04T16:04:08.761418+0000","last_active":"2024-03-04T16:04:08.761398+0000","last_peered":"2024-03-04T16:04:08.761398+0000","last_clean":"2024-03-04T16:04:08.761398+0000","last_became_active":"2024-03-04T16:02:43.716542+0000","last_became_peered":"2024-03-04T16:02:43.716542+0000","last_unstale":"2024-03-04T16:04:08.761398+0000","last_undegraded":"2024-03-04T16:04:08.761398+0000","last_fullsized":"2024-03-04T16:04:08.761398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.761341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.761341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:10.153431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095860+0000","last_change":"2024-03-04T16:02:45.754181+0000","last_active":"2024-03-04T16:04:00.095860+0000","last_peered":"2024-03-04T16:04:00.095860+0000","last_clean":"2024-03-04T16:04:00.095860+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:04:00.095860+0000","last_undegraded":"2024-03-04T16:04:00.095860+0000","last_fullsized":"2024-03-04T16:04:00.095860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.606250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096015+0000","last_change":"2024-03-04T16:02:49.791755+0000","last_active":"2024-03-04T16:04:00.096015+0000","last_peered":"2024-03-04T16:04:00.096015+0000","last_clean":"2024-03-04T16:04:00.096015+0000","last_became_active":"2024-03-04T16:02:49.791216+0000","last_became_peered":"2024-03-04T16:02:49.791216+0000","last_unstale":"2024-03-04T16:04:00.096015+0000","last_undegraded":"2024-03-04T16:04:00.096015+0000","last_fullsized":"2024-03-04T16:04:00.096015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.261130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171216+0000","last_change":"2024-03-04T16:02:47.773727+0000","last_active":"2024-03-04T16:04:00.171216+0000","last_peered":"2024-03-04T16:04:00.171216+0000","last_clean":"2024-03-04T16:04:00.171216+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:04:00.171216+0000","last_undegraded":"2024-03-04T16:04:00.171216+0000","last_fullsized":"2024-03-04T16:04:00.171216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.258206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171414+0000","last_change":"2024-03-04T16:03:52.330168+0000","last_active":"2024-03-04T16:04:00.171414+0000","last_peered":"2024-03-04T16:04:00.171414+0000","last_clean":"2024-03-04T16:04:00.171414+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:04:00.171414+0000","last_undegraded":"2024-03-04T16:04:00.171414+0000","last_fullsized":"2024-03-04T16:04:00.171414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.330026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.330026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.392092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096129+0000","last_change":"2024-03-04T16:02:45.753666+0000","last_active":"2024-03-04T16:04:00.096129+0000","last_peered":"2024-03-04T16:04:00.096129+0000","last_clean":"2024-03-04T16:04:00.096129+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:04:00.096129+0000","last_undegraded":"2024-03-04T16:04:00.096129+0000","last_fullsized":"2024-03-04T16:04:00.096129+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.790980+0000","last_change":"2024-03-04T16:04:03.791013+0000","last_active":"2024-03-04T16:04:03.790980+0000","last_peered":"2024-03-04T16:04:03.790980+0000","last_clean":"2024-03-04T16:04:03.790980+0000","last_became_active":"2024-03-04T16:02:49.794197+0000","last_became_peered":"2024-03-04T16:02:49.794197+0000","last_unstale":"2024-03-04T16:04:03.790980+0000","last_undegraded":"2024-03-04T16:04:03.790980+0000","last_fullsized":"2024-03-04T16:04:03.790980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.790905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.790905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.304942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096301+0000","last_change":"2024-03-04T16:02:47.777366+0000","last_active":"2024-03-04T16:04:00.096301+0000","last_peered":"2024-03-04T16:04:00.096301+0000","last_clean":"2024-03-04T16:04:00.096301+0000","last_became_active":"2024-03-04T16:02:47.777008+0000","last_became_peered":"2024-03-04T16:02:47.777008+0000","last_unstale":"2024-03-04T16:04:00.096301+0000","last_undegraded":"2024-03-04T16:04:00.096301+0000","last_fullsized":"2024-03-04T16:04:00.096301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.335199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172131+0000","last_change":"2024-03-04T16:02:43.711547+0000","last_active":"2024-03-04T16:04:00.172131+0000","last_peered":"2024-03-04T16:04:00.172131+0000","last_clean":"2024-03-04T16:04:00.172131+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:04:00.172131+0000","last_undegraded":"2024-03-04T16:04:00.172131+0000","last_fullsized":"2024-03-04T16:04:00.172131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.527444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.040200+0000","last_change":"2024-03-04T16:04:07.040237+0000","last_active":"2024-03-04T16:04:07.040200+0000","last_peered":"2024-03-04T16:04:07.040200+0000","last_clean":"2024-03-04T16:04:07.040200+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:04:07.040200+0000","last_undegraded":"2024-03-04T16:04:07.040200+0000","last_fullsized":"2024-03-04T16:04:07.040200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.040136+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.386047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095861+0000","last_change":"2024-03-04T16:02:49.792155+0000","last_active":"2024-03-04T16:04:00.095861+0000","last_peered":"2024-03-04T16:04:00.095861+0000","last_clean":"2024-03-04T16:04:00.095861+0000","last_became_active":"2024-03-04T16:02:49.791815+0000","last_became_peered":"2024-03-04T16:02:49.791815+0000","last_unstale":"2024-03-04T16:04:00.095861+0000","last_undegraded":"2024-03-04T16:04:00.095861+0000","last_fullsized":"2024-03-04T16:04:00.095861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.727768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170892+0000","last_change":"2024-03-04T16:03:51.939795+0000","last_active":"2024-03-04T16:04:00.170892+0000","last_peered":"2024-03-04T16:04:00.170892+0000","last_clean":"2024-03-04T16:04:00.170892+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:04:00.170892+0000","last_undegraded":"2024-03-04T16:04:00.170892+0000","last_fullsized":"2024-03-04T16:04:00.170892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.939712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.939712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.441874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171382+0000","last_change":"2024-03-04T16:03:44.419164+0000","last_active":"2024-03-04T16:04:00.171382+0000","last_peered":"2024-03-04T16:04:00.171382+0000","last_clean":"2024-03-04T16:04:00.171382+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:04:00.171382+0000","last_undegraded":"2024-03-04T16:04:00.171382+0000","last_fullsized":"2024-03-04T16:04:00.171382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:44.419065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:44.419065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.867222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171695+0000","last_change":"2024-03-04T16:03:50.400887+0000","last_active":"2024-03-04T16:04:00.171695+0000","last_peered":"2024-03-04T16:04:00.171695+0000","last_clean":"2024-03-04T16:04:00.171695+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:04:00.171695+0000","last_undegraded":"2024-03-04T16:04:00.171695+0000","last_fullsized":"2024-03-04T16:04:00.171695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:50.400784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.400784+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.749407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171573+0000","last_change":"2024-03-04T16:02:49.797503+0000","last_active":"2024-03-04T16:04:00.171573+0000","last_peered":"2024-03-04T16:04:00.171573+0000","last_clean":"2024-03-04T16:04:00.171573+0000","last_became_active":"2024-03-04T16:02:49.797174+0000","last_became_peered":"2024-03-04T16:02:49.797174+0000","last_unstale":"2024-03-04T16:04:00.171573+0000","last_undegraded":"2024-03-04T16:04:00.171573+0000","last_fullsized":"2024-03-04T16:04:00.171573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:17.229451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.217113+0000","last_change":"2024-03-04T16:04:10.217159+0000","last_active":"2024-03-04T16:04:10.217113+0000","last_peered":"2024-03-04T16:04:10.217113+0000","last_clean":"2024-03-04T16:04:10.217113+0000","last_became_active":"2024-03-04T16:02:47.768874+0000","last_became_peered":"2024-03-04T16:02:47.768874+0000","last_unstale":"2024-03-04T16:04:10.217113+0000","last_undegraded":"2024-03-04T16:04:10.217113+0000","last_fullsized":"2024-03-04T16:04:10.217113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:10.217024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.217024+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:39.681485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.837345+0000","last_change":"2024-03-04T16:04:06.837374+0000","last_active":"2024-03-04T16:04:06.837345+0000","last_peered":"2024-03-04T16:04:06.837345+0000","last_clean":"2024-03-04T16:04:06.837345+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:04:06.837345+0000","last_undegraded":"2024-03-04T16:04:06.837345+0000","last_fullsized":"2024-03-04T16:04:06.837345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:06.837263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.837263+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.977815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096389+0000","last_change":"2024-03-04T16:02:45.749382+0000","last_active":"2024-03-04T16:04:00.096389+0000","last_peered":"2024-03-04T16:04:00.096389+0000","last_clean":"2024-03-04T16:04:00.096389+0000","last_became_active":"2024-03-04T16:02:45.748824+0000","last_became_peered":"2024-03-04T16:02:45.748824+0000","last_unstale":"2024-03-04T16:04:00.096389+0000","last_undegraded":"2024-03-04T16:04:00.096389+0000","last_fullsized":"2024-03-04T16:04:00.096389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.580370+0000","last_change":"2024-03-04T16:04:10.580391+0000","last_active":"2024-03-04T16:04:10.580370+0000","last_peered":"2024-03-04T16:04:10.580370+0000","last_clean":"2024-03-04T16:04:10.580370+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:04:10.580370+0000","last_undegraded":"2024-03-04T16:04:10.580370+0000","last_fullsized":"2024-03-04T16:04:10.580370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:10.580305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.580305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.834007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774579+0000","last_change":"2024-03-04T16:02:47.774221+0000","last_active":"2024-03-04T16:04:03.774579+0000","last_peered":"2024-03-04T16:04:03.774579+0000","last_clean":"2024-03-04T16:04:03.774579+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:04:03.774579+0000","last_undegraded":"2024-03-04T16:04:03.774579+0000","last_fullsized":"2024-03-04T16:04:03.774579+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170631+0000","last_change":"2024-03-04T16:03:49.810121+0000","last_active":"2024-03-04T16:04:00.170631+0000","last_peered":"2024-03-04T16:04:00.170631+0000","last_clean":"2024-03-04T16:04:00.170631+0000","last_became_active":"2024-03-04T16:02:43.713331+0000","last_became_peered":"2024-03-04T16:02:43.713331+0000","last_unstale":"2024-03-04T16:04:00.170631+0000","last_undegraded":"2024-03-04T16:04:00.170631+0000","last_fullsized":"2024-03-04T16:04:00.170631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:03:49.810002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:49.810002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.948952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172670+0000","last_change":"2024-03-04T16:03:55.306308+0000","last_active":"2024-03-04T16:04:00.172670+0000","last_peered":"2024-03-04T16:04:00.172670+0000","last_clean":"2024-03-04T16:04:00.172670+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:04:00.172670+0000","last_undegraded":"2024-03-04T16:04:00.172670+0000","last_fullsized":"2024-03-04T16:04:00.172670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:03:55.306180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.306180+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.830154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095503+0000","last_change":"2024-03-04T16:02:49.790101+0000","last_active":"2024-03-04T16:04:00.095503+0000","last_peered":"2024-03-04T16:04:00.095503+0000","last_clean":"2024-03-04T16:04:00.095503+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:04:00.095503+0000","last_undegraded":"2024-03-04T16:04:00.095503+0000","last_fullsized":"2024-03-04T16:04:00.095503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:02.704953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096828+0000","last_change":"2024-03-04T16:02:47.775318+0000","last_active":"2024-03-04T16:04:00.096828+0000","last_peered":"2024-03-04T16:04:00.096828+0000","last_clean":"2024-03-04T16:04:00.096828+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:04:00.096828+0000","last_undegraded":"2024-03-04T16:04:00.096828+0000","last_fullsized":"2024-03-04T16:04:00.096828+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171544+0000","last_change":"2024-03-04T16:02:43.714055+0000","last_active":"2024-03-04T16:04:00.171544+0000","last_peered":"2024-03-04T16:04:00.171544+0000","last_clean":"2024-03-04T16:04:00.171544+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:04:00.171544+0000","last_undegraded":"2024-03-04T16:04:00.171544+0000","last_fullsized":"2024-03-04T16:04:00.171544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.115403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.755480+0000","last_change":"2024-03-04T16:04:05.755536+0000","last_active":"2024-03-04T16:04:05.755480+0000","last_peered":"2024-03-04T16:04:05.755480+0000","last_clean":"2024-03-04T16:04:05.755480+0000","last_became_active":"2024-03-04T16:02:45.745571+0000","last_became_peered":"2024-03-04T16:02:45.745571+0000","last_unstale":"2024-03-04T16:04:05.755480+0000","last_undegraded":"2024-03-04T16:04:05.755480+0000","last_fullsized":"2024-03-04T16:04:05.755480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:05.755370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.755370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.469566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.871295+0000","last_change":"2024-03-04T16:04:09.871332+0000","last_active":"2024-03-04T16:04:09.871295+0000","last_peered":"2024-03-04T16:04:09.871295+0000","last_clean":"2024-03-04T16:04:09.871295+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:04:09.871295+0000","last_undegraded":"2024-03-04T16:04:09.871295+0000","last_fullsized":"2024-03-04T16:04:09.871295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.871214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.871214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:30.539294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:11.175170+0000","last_change":"2024-03-04T16:04:11.175190+0000","last_active":"2024-03-04T16:04:11.175170+0000","last_peered":"2024-03-04T16:04:11.175170+0000","last_clean":"2024-03-04T16:04:11.175170+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:04:11.175170+0000","last_undegraded":"2024-03-04T16:04:11.175170+0000","last_fullsized":"2024-03-04T16:04:11.175170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:11.175113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:11.175113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.876575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.086907+0000","last_change":"2024-03-04T16:03:46.718298+0000","last_active":"2024-03-04T16:04:00.086907+0000","last_peered":"2024-03-04T16:04:00.086907+0000","last_clean":"2024-03-04T16:04:00.086907+0000","last_became_active":"2024-03-04T16:02:43.718241+0000","last_became_peered":"2024-03-04T16:02:43.718241+0000","last_unstale":"2024-03-04T16:04:00.086907+0000","last_undegraded":"2024-03-04T16:04:00.086907+0000","last_fullsized":"2024-03-04T16:04:00.086907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.718188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.606455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096922+0000","last_change":"2024-03-04T16:03:52.421377+0000","last_active":"2024-03-04T16:04:00.096922+0000","last_peered":"2024-03-04T16:04:00.096922+0000","last_clean":"2024-03-04T16:04:00.096922+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:04:00.096922+0000","last_undegraded":"2024-03-04T16:04:00.096922+0000","last_fullsized":"2024-03-04T16:04:00.096922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.421285+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.703150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171188+0000","last_change":"2024-03-04T16:03:58.313071+0000","last_active":"2024-03-04T16:04:00.171188+0000","last_peered":"2024-03-04T16:04:00.171188+0000","last_clean":"2024-03-04T16:04:00.171188+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:04:00.171188+0000","last_undegraded":"2024-03-04T16:04:00.171188+0000","last_fullsized":"2024-03-04T16:04:00.171188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.312925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.312925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.584207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172545+0000","last_change":"2024-03-04T16:02:43.717223+0000","last_active":"2024-03-04T16:04:00.172545+0000","last_peered":"2024-03-04T16:04:00.172545+0000","last_clean":"2024-03-04T16:04:00.172545+0000","last_became_active":"2024-03-04T16:02:43.716945+0000","last_became_peered":"2024-03-04T16:02:43.716945+0000","last_unstale":"2024-03-04T16:04:00.172545+0000","last_undegraded":"2024-03-04T16:04:00.172545+0000","last_fullsized":"2024-03-04T16:04:00.172545+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.985563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170668+0000","last_change":"2024-03-04T16:03:48.260185+0000","last_active":"2024-03-04T16:04:00.170668+0000","last_peered":"2024-03-04T16:04:00.170668+0000","last_clean":"2024-03-04T16:04:00.170668+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:04:00.170668+0000","last_undegraded":"2024-03-04T16:04:00.170668+0000","last_fullsized":"2024-03-04T16:04:00.170668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:48.260047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.260047+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.815204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774690+0000","last_change":"2024-03-04T16:02:47.774009+0000","last_active":"2024-03-04T16:04:03.774690+0000","last_peered":"2024-03-04T16:04:03.774690+0000","last_clean":"2024-03-04T16:04:03.774690+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:04:03.774690+0000","last_undegraded":"2024-03-04T16:04:03.774690+0000","last_fullsized":"2024-03-04T16:04:03.774690+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.594457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096118+0000","last_change":"2024-03-04T16:02:49.796054+0000","last_active":"2024-03-04T16:04:00.096118+0000","last_peered":"2024-03-04T16:04:00.096118+0000","last_clean":"2024-03-04T16:04:00.096118+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:04:00.096118+0000","last_undegraded":"2024-03-04T16:04:00.096118+0000","last_fullsized":"2024-03-04T16:04:00.096118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.821795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172343+0000","last_change":"2024-03-04T16:03:55.817828+0000","last_active":"2024-03-04T16:04:00.172343+0000","last_peered":"2024-03-04T16:04:00.172343+0000","last_clean":"2024-03-04T16:04:00.172343+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:04:00.172343+0000","last_undegraded":"2024-03-04T16:04:00.172343+0000","last_fullsized":"2024-03-04T16:04:00.172343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:55.817709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.817709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.939046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171220+0000","last_change":"2024-03-04T16:03:48.407212+0000","last_active":"2024-03-04T16:04:00.171220+0000","last_peered":"2024-03-04T16:04:00.171220+0000","last_clean":"2024-03-04T16:04:00.171220+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:04:00.171220+0000","last_undegraded":"2024-03-04T16:04:00.171220+0000","last_fullsized":"2024-03-04T16:04:00.171220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:48.407088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.407088+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:49.238194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095123+0000","last_change":"2024-03-04T16:03:59.048702+0000","last_active":"2024-03-04T16:04:00.095123+0000","last_peered":"2024-03-04T16:04:00.095123+0000","last_clean":"2024-03-04T16:04:00.095123+0000","last_became_active":"2024-03-04T16:02:47.767316+0000","last_became_peered":"2024-03-04T16:02:47.767316+0000","last_unstale":"2024-03-04T16:04:00.095123+0000","last_undegraded":"2024-03-04T16:04:00.095123+0000","last_fullsized":"2024-03-04T16:04:00.095123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:04.396312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170696+0000","last_change":"2024-03-04T16:02:49.797192+0000","last_active":"2024-03-04T16:04:00.170696+0000","last_peered":"2024-03-04T16:04:00.170696+0000","last_clean":"2024-03-04T16:04:00.170696+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:04:00.170696+0000","last_undegraded":"2024-03-04T16:04:00.170696+0000","last_fullsized":"2024-03-04T16:04:00.170696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.448969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095320+0000","last_change":"2024-03-04T16:03:51.984808+0000","last_active":"2024-03-04T16:04:00.095320+0000","last_peered":"2024-03-04T16:04:00.095320+0000","last_clean":"2024-03-04T16:04:00.095320+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:04:00.095320+0000","last_undegraded":"2024-03-04T16:04:00.095320+0000","last_fullsized":"2024-03-04T16:04:00.095320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.984721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.984721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.092310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172295+0000","last_change":"2024-03-04T16:02:43.715496+0000","last_active":"2024-03-04T16:04:00.172295+0000","last_peered":"2024-03-04T16:04:00.172295+0000","last_clean":"2024-03-04T16:04:00.172295+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:04:00.172295+0000","last_undegraded":"2024-03-04T16:04:00.172295+0000","last_fullsized":"2024-03-04T16:04:00.172295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.926799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087201+0000","last_change":"2024-03-04T16:02:45.747239+0000","last_active":"2024-03-04T16:04:00.087201+0000","last_peered":"2024-03-04T16:04:00.087201+0000","last_clean":"2024-03-04T16:04:00.087201+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:04:00.087201+0000","last_undegraded":"2024-03-04T16:04:00.087201+0000","last_fullsized":"2024-03-04T16:04:00.087201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.287130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170394+0000","last_change":"2024-03-04T16:03:50.786884+0000","last_active":"2024-03-04T16:04:00.170394+0000","last_peered":"2024-03-04T16:04:00.170394+0000","last_clean":"2024-03-04T16:04:00.170394+0000","last_became_active":"2024-03-04T16:02:49.793023+0000","last_became_peered":"2024-03-04T16:02:49.793023+0000","last_unstale":"2024-03-04T16:04:00.170394+0000","last_undegraded":"2024-03-04T16:04:00.170394+0000","last_fullsized":"2024-03-04T16:04:00.170394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.786805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.786805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.570842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095255+0000","last_change":"2024-03-04T16:02:47.769558+0000","last_active":"2024-03-04T16:04:00.095255+0000","last_peered":"2024-03-04T16:04:00.095255+0000","last_clean":"2024-03-04T16:04:00.095255+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:04:00.095255+0000","last_undegraded":"2024-03-04T16:04:00.095255+0000","last_fullsized":"2024-03-04T16:04:00.095255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.736215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.230097+0000","last_change":"2024-03-04T16:04:06.230119+0000","last_active":"2024-03-04T16:04:06.230097+0000","last_peered":"2024-03-04T16:04:06.230097+0000","last_clean":"2024-03-04T16:04:06.230097+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:04:06.230097+0000","last_undegraded":"2024-03-04T16:04:06.230097+0000","last_fullsized":"2024-03-04T16:04:06.230097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.230030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.804736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171391+0000","last_change":"2024-03-04T16:02:45.752568+0000","last_active":"2024-03-04T16:04:00.171391+0000","last_peered":"2024-03-04T16:04:00.171391+0000","last_clean":"2024-03-04T16:04:00.171391+0000","last_became_active":"2024-03-04T16:02:45.752201+0000","last_became_peered":"2024-03-04T16:02:45.752201+0000","last_unstale":"2024-03-04T16:04:00.171391+0000","last_undegraded":"2024-03-04T16:04:00.171391+0000","last_fullsized":"2024-03-04T16:04:00.171391+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.531681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096551+0000","last_change":"2024-03-04T16:02:49.793485+0000","last_active":"2024-03-04T16:04:00.096551+0000","last_peered":"2024-03-04T16:04:00.096551+0000","last_clean":"2024-03-04T16:04:00.096551+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:04:00.096551+0000","last_undegraded":"2024-03-04T16:04:00.096551+0000","last_fullsized":"2024-03-04T16:04:00.096551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.099839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170930+0000","last_change":"2024-03-04T16:03:54.344325+0000","last_active":"2024-03-04T16:04:00.170930+0000","last_peered":"2024-03-04T16:04:00.170930+0000","last_clean":"2024-03-04T16:04:00.170930+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:04:00.170930+0000","last_undegraded":"2024-03-04T16:04:00.170930+0000","last_fullsized":"2024-03-04T16:04:00.170930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:03:54.344193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:54.344193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.913538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171536+0000","last_change":"2024-03-04T16:03:59.437627+0000","last_active":"2024-03-04T16:04:00.171536+0000","last_peered":"2024-03-04T16:04:00.171536+0000","last_clean":"2024-03-04T16:04:00.171536+0000","last_became_active":"2024-03-04T16:02:43.717129+0000","last_became_peered":"2024-03-04T16:02:43.717129+0000","last_unstale":"2024-03-04T16:04:00.171536+0000","last_undegraded":"2024-03-04T16:04:00.171536+0000","last_fullsized":"2024-03-04T16:04:00.171536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.437524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.437524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.171391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":117,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774688+0000","last_change":"2024-03-04T16:02:45.753468+0000","last_active":"2024-03-04T16:04:03.774688+0000","last_peered":"2024-03-04T16:04:03.774688+0000","last_clean":"2024-03-04T16:04:03.774688+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:04:03.774688+0000","last_undegraded":"2024-03-04T16:04:03.774688+0000","last_fullsized":"2024-03-04T16:04:03.774688+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.486508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171374+0000","last_change":"2024-03-04T16:02:47.773975+0000","last_active":"2024-03-04T16:04:00.171374+0000","last_peered":"2024-03-04T16:04:00.171374+0000","last_clean":"2024-03-04T16:04:00.171374+0000","last_became_active":"2024-03-04T16:02:47.773547+0000","last_became_peered":"2024-03-04T16:02:47.773547+0000","last_unstale":"2024-03-04T16:04:00.171374+0000","last_undegraded":"2024-03-04T16:04:00.171374+0000","last_fullsized":"2024-03-04T16:04:00.171374+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172366+0000","last_change":"2024-03-04T16:02:49.791079+0000","last_active":"2024-03-04T16:04:00.172366+0000","last_peered":"2024-03-04T16:04:00.172366+0000","last_clean":"2024-03-04T16:04:00.172366+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:04:00.172366+0000","last_undegraded":"2024-03-04T16:04:00.172366+0000","last_fullsized":"2024-03-04T16:04:00.172366+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.701071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170995+0000","last_change":"2024-03-04T16:02:43.714877+0000","last_active":"2024-03-04T16:04:00.170995+0000","last_peered":"2024-03-04T16:04:00.170995+0000","last_clean":"2024-03-04T16:04:00.170995+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:04:00.170995+0000","last_undegraded":"2024-03-04T16:04:00.170995+0000","last_fullsized":"2024-03-04T16:04:00.170995+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095150+0000","last_change":"2024-03-04T16:03:47.979184+0000","last_active":"2024-03-04T16:04:00.095150+0000","last_peered":"2024-03-04T16:04:00.095150+0000","last_clean":"2024-03-04T16:04:00.095150+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:04:00.095150+0000","last_undegraded":"2024-03-04T16:04:00.095150+0000","last_fullsized":"2024-03-04T16:04:00.095150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.979058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.979058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.315882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"75'13","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171982+0000","last_change":"2024-03-04T16:02:45.748924+0000","last_active":"2024-03-04T16:04:00.171982+0000","last_peered":"2024-03-04T16:04:00.171982+0000","last_clean":"2024-03-04T16:04:00.171982+0000","last_became_active":"2024-03-04T16:02:45.748615+0000","last_became_peered":"2024-03-04T16:02:45.748615+0000","last_unstale":"2024-03-04T16:04:00.171982+0000","last_undegraded":"2024-03-04T16:04:00.171982+0000","last_fullsized":"2024-03-04T16:04:00.171982+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.429848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.048679+0000","last_change":"2024-03-04T16:04:03.048703+0000","last_active":"2024-03-04T16:04:03.048679+0000","last_peered":"2024-03-04T16:04:03.048679+0000","last_clean":"2024-03-04T16:04:03.048679+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:04:03.048679+0000","last_undegraded":"2024-03-04T16:04:03.048679+0000","last_fullsized":"2024-03-04T16:04:03.048679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.048620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.048620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.405283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172223+0000","last_change":"2024-03-04T16:02:49.794677+0000","last_active":"2024-03-04T16:04:00.172223+0000","last_peered":"2024-03-04T16:04:00.172223+0000","last_clean":"2024-03-04T16:04:00.172223+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:04:00.172223+0000","last_undegraded":"2024-03-04T16:04:00.172223+0000","last_fullsized":"2024-03-04T16:04:00.172223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.070330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170921+0000","last_change":"2024-03-04T16:03:12.851794+0000","last_active":"2024-03-04T16:04:00.170921+0000","last_peered":"2024-03-04T16:04:00.170921+0000","last_clean":"2024-03-04T16:04:00.170921+0000","last_became_active":"2024-03-04T16:02:34.079164+0000","last_became_peered":"2024-03-04T16:02:34.079164+0000","last_unstale":"2024-03-04T16:04:00.170921+0000","last_undegraded":"2024-03-04T16:04:00.170921+0000","last_fullsized":"2024-03-04T16:04:00.170921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'119","last_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:03:12.851677+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:36.356853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096256+0000","last_change":"2024-03-04T16:03:57.445763+0000","last_active":"2024-03-04T16:04:00.096256+0000","last_peered":"2024-03-04T16:04:00.096256+0000","last_clean":"2024-03-04T16:04:00.096256+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:04:00.096256+0000","last_undegraded":"2024-03-04T16:04:00.096256+0000","last_fullsized":"2024-03-04T16:04:00.096256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.445659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.445659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.402395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":99,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774961+0000","last_change":"2024-03-04T16:03:58.764928+0000","last_active":"2024-03-04T16:04:03.774961+0000","last_peered":"2024-03-04T16:04:03.774961+0000","last_clean":"2024-03-04T16:04:03.774961+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:04:03.774961+0000","last_undegraded":"2024-03-04T16:04:03.774961+0000","last_fullsized":"2024-03-04T16:04:03.774961+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:03:58.764821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.764821+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.352281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172055+0000","last_change":"2024-03-04T16:03:52.917188+0000","last_active":"2024-03-04T16:04:00.172055+0000","last_peered":"2024-03-04T16:04:00.172055+0000","last_clean":"2024-03-04T16:04:00.172055+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:04:00.172055+0000","last_undegraded":"2024-03-04T16:04:00.172055+0000","last_fullsized":"2024-03-04T16:04:00.172055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.917090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.917090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:00.018319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171270+0000","last_change":"2024-03-04T16:03:52.944487+0000","last_active":"2024-03-04T16:04:00.171270+0000","last_peered":"2024-03-04T16:04:00.171270+0000","last_clean":"2024-03-04T16:04:00.171270+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:04:00.171270+0000","last_undegraded":"2024-03-04T16:04:00.171270+0000","last_fullsized":"2024-03-04T16:04:00.171270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.944410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.132708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.738565+0000","last_change":"2024-03-04T16:04:09.738590+0000","last_active":"2024-03-04T16:04:09.738565+0000","last_peered":"2024-03-04T16:04:09.738565+0000","last_clean":"2024-03-04T16:04:09.738565+0000","last_became_active":"2024-03-04T16:02:43.713276+0000","last_became_peered":"2024-03-04T16:02:43.713276+0000","last_unstale":"2024-03-04T16:04:09.738565+0000","last_undegraded":"2024-03-04T16:04:09.738565+0000","last_fullsized":"2024-03-04T16:04:09.738565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.738490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.738490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.085127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":158,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:11.142612+0000","last_change":"2024-03-04T16:04:07.872620+0000","last_active":"2024-03-04T16:04:11.142612+0000","last_peered":"2024-03-04T16:04:11.142612+0000","last_clean":"2024-03-04T16:04:11.142612+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:04:11.142612+0000","last_undegraded":"2024-03-04T16:04:11.142612+0000","last_fullsized":"2024-03-04T16:04:11.142612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-04T16:04:07.872489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.872489+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.219006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171663+0000","last_change":"2024-03-04T16:02:45.755785+0000","last_active":"2024-03-04T16:04:00.171663+0000","last_peered":"2024-03-04T16:04:00.171663+0000","last_clean":"2024-03-04T16:04:00.171663+0000","last_became_active":"2024-03-04T16:02:45.755435+0000","last_became_peered":"2024-03-04T16:02:45.755435+0000","last_unstale":"2024-03-04T16:04:00.171663+0000","last_undegraded":"2024-03-04T16:04:00.171663+0000","last_fullsized":"2024-03-04T16:04:00.171663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.974009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172637+0000","last_change":"2024-03-04T16:03:57.296517+0000","last_active":"2024-03-04T16:04:00.172637+0000","last_peered":"2024-03-04T16:04:00.172637+0000","last_clean":"2024-03-04T16:04:00.172637+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:04:00.172637+0000","last_undegraded":"2024-03-04T16:04:00.172637+0000","last_fullsized":"2024-03-04T16:04:00.172637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.296421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.296421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.272283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170706+0000","last_change":"2024-03-04T16:02:49.785263+0000","last_active":"2024-03-04T16:04:00.170706+0000","last_peered":"2024-03-04T16:04:00.170706+0000","last_clean":"2024-03-04T16:04:00.170706+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:04:00.170706+0000","last_undegraded":"2024-03-04T16:04:00.170706+0000","last_fullsized":"2024-03-04T16:04:00.170706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171392+0000","last_change":"2024-03-04T16:02:43.719367+0000","last_active":"2024-03-04T16:04:00.171392+0000","last_peered":"2024-03-04T16:04:00.171392+0000","last_clean":"2024-03-04T16:04:00.171392+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:04:00.171392+0000","last_undegraded":"2024-03-04T16:04:00.171392+0000","last_fullsized":"2024-03-04T16:04:00.171392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.746292+0000","last_change":"2024-03-04T16:04:10.746331+0000","last_active":"2024-03-04T16:04:10.746292+0000","last_peered":"2024-03-04T16:04:10.746292+0000","last_clean":"2024-03-04T16:04:10.746292+0000","last_became_active":"2024-03-04T16:02:45.744172+0000","last_became_peered":"2024-03-04T16:02:45.744172+0000","last_unstale":"2024-03-04T16:04:10.746292+0000","last_undegraded":"2024-03-04T16:04:10.746292+0000","last_fullsized":"2024-03-04T16:04:10.746292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:04:10.746233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.746233+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:36.591472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087204+0000","last_change":"2024-03-04T16:03:50.770141+0000","last_active":"2024-03-04T16:04:00.087204+0000","last_peered":"2024-03-04T16:04:00.087204+0000","last_clean":"2024-03-04T16:04:00.087204+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:04:00.087204+0000","last_undegraded":"2024-03-04T16:04:00.087204+0000","last_fullsized":"2024-03-04T16:04:00.087204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.770048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.770048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.417482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170347+0000","last_change":"2024-03-04T16:02:49.797759+0000","last_active":"2024-03-04T16:04:00.170347+0000","last_peered":"2024-03-04T16:04:00.170347+0000","last_clean":"2024-03-04T16:04:00.170347+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:04:00.170347+0000","last_undegraded":"2024-03-04T16:04:00.170347+0000","last_fullsized":"2024-03-04T16:04:00.170347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.105311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172561+0000","last_change":"2024-03-04T16:02:43.711336+0000","last_active":"2024-03-04T16:04:00.172561+0000","last_peered":"2024-03-04T16:04:00.172561+0000","last_clean":"2024-03-04T16:04:00.172561+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:04:00.172561+0000","last_undegraded":"2024-03-04T16:04:00.172561+0000","last_fullsized":"2024-03-04T16:04:00.172561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:11.216071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170915+0000","last_change":"2024-03-04T16:02:45.748912+0000","last_active":"2024-03-04T16:04:00.170915+0000","last_peered":"2024-03-04T16:04:00.170915+0000","last_clean":"2024-03-04T16:04:00.170915+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:04:00.170915+0000","last_undegraded":"2024-03-04T16:04:00.170915+0000","last_fullsized":"2024-03-04T16:04:00.170915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774537+0000","last_change":"2024-03-04T16:02:47.777918+0000","last_active":"2024-03-04T16:04:03.774537+0000","last_peered":"2024-03-04T16:04:03.774537+0000","last_clean":"2024-03-04T16:04:03.774537+0000","last_became_active":"2024-03-04T16:02:47.777462+0000","last_became_peered":"2024-03-04T16:02:47.777462+0000","last_unstale":"2024-03-04T16:04:03.774537+0000","last_undegraded":"2024-03-04T16:04:03.774537+0000","last_fullsized":"2024-03-04T16:04:03.774537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.300423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171829+0000","last_change":"2024-03-04T16:03:56.858487+0000","last_active":"2024-03-04T16:04:00.171829+0000","last_peered":"2024-03-04T16:04:00.171829+0000","last_clean":"2024-03-04T16:04:00.171829+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:04:00.171829+0000","last_undegraded":"2024-03-04T16:04:00.171829+0000","last_fullsized":"2024-03-04T16:04:00.171829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.858376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.858376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.769864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":56,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171668+0000","last_change":"2024-03-04T16:02:43.712924+0000","last_active":"2024-03-04T16:04:00.171668+0000","last_peered":"2024-03-04T16:04:00.171668+0000","last_clean":"2024-03-04T16:04:00.171668+0000","last_became_active":"2024-03-04T16:02:43.712287+0000","last_became_peered":"2024-03-04T16:02:43.712287+0000","last_unstale":"2024-03-04T16:04:00.171668+0000","last_undegraded":"2024-03-04T16:04:00.171668+0000","last_fullsized":"2024-03-04T16:04:00.171668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.101366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.515895+0000","last_change":"2024-03-04T16:04:06.515940+0000","last_active":"2024-03-04T16:04:06.515895+0000","last_peered":"2024-03-04T16:04:06.515895+0000","last_clean":"2024-03-04T16:04:06.515895+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:04:06.515895+0000","last_undegraded":"2024-03-04T16:04:06.515895+0000","last_fullsized":"2024-03-04T16:04:06.515895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.515823+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.602986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.344525+0000","last_change":"2024-03-04T16:04:09.344572+0000","last_active":"2024-03-04T16:04:09.344525+0000","last_peered":"2024-03-04T16:04:09.344525+0000","last_clean":"2024-03-04T16:04:09.344525+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:04:09.344525+0000","last_undegraded":"2024-03-04T16:04:09.344525+0000","last_fullsized":"2024-03-04T16:04:09.344525+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.344448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.344448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.358656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171503+0000","last_change":"2024-03-04T16:02:49.789412+0000","last_active":"2024-03-04T16:04:00.171503+0000","last_peered":"2024-03-04T16:04:00.171503+0000","last_clean":"2024-03-04T16:04:00.171503+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:04:00.171503+0000","last_undegraded":"2024-03-04T16:04:00.171503+0000","last_fullsized":"2024-03-04T16:04:00.171503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.799791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096671+0000","last_change":"2024-03-04T16:02:43.716171+0000","last_active":"2024-03-04T16:04:00.096671+0000","last_peered":"2024-03-04T16:04:00.096671+0000","last_clean":"2024-03-04T16:04:00.096671+0000","last_became_active":"2024-03-04T16:02:43.715346+0000","last_became_peered":"2024-03-04T16:02:43.715346+0000","last_unstale":"2024-03-04T16:04:00.096671+0000","last_undegraded":"2024-03-04T16:04:00.096671+0000","last_fullsized":"2024-03-04T16:04:00.096671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095990+0000","last_change":"2024-03-04T16:02:45.749977+0000","last_active":"2024-03-04T16:04:00.095990+0000","last_peered":"2024-03-04T16:04:00.095990+0000","last_clean":"2024-03-04T16:04:00.095990+0000","last_became_active":"2024-03-04T16:02:45.749633+0000","last_became_peered":"2024-03-04T16:02:45.749633+0000","last_unstale":"2024-03-04T16:04:00.095990+0000","last_undegraded":"2024-03-04T16:04:00.095990+0000","last_fullsized":"2024-03-04T16:04:00.095990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.646959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170756+0000","last_change":"2024-03-04T16:02:47.768154+0000","last_active":"2024-03-04T16:04:00.170756+0000","last_peered":"2024-03-04T16:04:00.170756+0000","last_clean":"2024-03-04T16:04:00.170756+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:04:00.170756+0000","last_undegraded":"2024-03-04T16:04:00.170756+0000","last_fullsized":"2024-03-04T16:04:00.170756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.581920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172282+0000","last_change":"2024-03-04T16:02:49.792150+0000","last_active":"2024-03-04T16:04:00.172282+0000","last_peered":"2024-03-04T16:04:00.172282+0000","last_clean":"2024-03-04T16:04:00.172282+0000","last_became_active":"2024-03-04T16:02:49.791851+0000","last_became_peered":"2024-03-04T16:02:49.791851+0000","last_unstale":"2024-03-04T16:04:00.172282+0000","last_undegraded":"2024-03-04T16:04:00.172282+0000","last_fullsized":"2024-03-04T16:04:00.172282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.262236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774705+0000","last_change":"2024-03-04T16:02:47.768859+0000","last_active":"2024-03-04T16:04:03.774705+0000","last_peered":"2024-03-04T16:04:03.774705+0000","last_clean":"2024-03-04T16:04:03.774705+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:04:03.774705+0000","last_undegraded":"2024-03-04T16:04:03.774705+0000","last_fullsized":"2024-03-04T16:04:03.774705+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.459494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095980+0000","last_change":"2024-03-04T16:03:58.462106+0000","last_active":"2024-03-04T16:04:00.095980+0000","last_peered":"2024-03-04T16:04:00.095980+0000","last_clean":"2024-03-04T16:04:00.095980+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:04:00.095980+0000","last_undegraded":"2024-03-04T16:04:00.095980+0000","last_fullsized":"2024-03-04T16:04:00.095980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.462018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:08.477439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171988+0000","last_change":"2024-03-04T16:03:46.963168+0000","last_active":"2024-03-04T16:04:00.171988+0000","last_peered":"2024-03-04T16:04:00.171988+0000","last_clean":"2024-03-04T16:04:00.171988+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:04:00.171988+0000","last_undegraded":"2024-03-04T16:04:00.171988+0000","last_fullsized":"2024-03-04T16:04:00.171988+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:46.963065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.963065+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.801631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096277+0000","last_change":"2024-03-04T16:02:49.794003+0000","last_active":"2024-03-04T16:04:00.096277+0000","last_peered":"2024-03-04T16:04:00.096277+0000","last_clean":"2024-03-04T16:04:00.096277+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:04:00.096277+0000","last_undegraded":"2024-03-04T16:04:00.096277+0000","last_fullsized":"2024-03-04T16:04:00.096277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.636557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172548+0000","last_change":"2024-03-04T16:02:47.766009+0000","last_active":"2024-03-04T16:04:00.172548+0000","last_peered":"2024-03-04T16:04:00.172548+0000","last_clean":"2024-03-04T16:04:00.172548+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:04:00.172548+0000","last_undegraded":"2024-03-04T16:04:00.172548+0000","last_fullsized":"2024-03-04T16:04:00.172548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.260597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170870+0000","last_change":"2024-03-04T16:02:43.705402+0000","last_active":"2024-03-04T16:04:00.170870+0000","last_peered":"2024-03-04T16:04:00.170870+0000","last_clean":"2024-03-04T16:04:00.170870+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:04:00.170870+0000","last_undegraded":"2024-03-04T16:04:00.170870+0000","last_fullsized":"2024-03-04T16:04:00.170870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.029120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172915+0000","last_change":"2024-03-04T16:02:45.746393+0000","last_active":"2024-03-04T16:04:00.172915+0000","last_peered":"2024-03-04T16:04:00.172915+0000","last_clean":"2024-03-04T16:04:00.172915+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:04:00.172915+0000","last_undegraded":"2024-03-04T16:04:00.172915+0000","last_fullsized":"2024-03-04T16:04:00.172915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.554587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.063138+0000","last_change":"2024-03-04T16:04:07.063161+0000","last_active":"2024-03-04T16:04:07.063138+0000","last_peered":"2024-03-04T16:04:07.063138+0000","last_clean":"2024-03-04T16:04:07.063138+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:04:07.063138+0000","last_undegraded":"2024-03-04T16:04:07.063138+0000","last_fullsized":"2024-03-04T16:04:07.063138+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.063063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.063063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.152553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.976020+0000","last_change":"2024-03-04T16:04:04.976057+0000","last_active":"2024-03-04T16:04:04.976020+0000","last_peered":"2024-03-04T16:04:04.976020+0000","last_clean":"2024-03-04T16:04:04.976020+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:04:04.976020+0000","last_undegraded":"2024-03-04T16:04:04.976020+0000","last_fullsized":"2024-03-04T16:04:04.976020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.975944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.975944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.116230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.763022+0000","last_change":"2024-03-04T16:04:07.763042+0000","last_active":"2024-03-04T16:04:07.763022+0000","last_peered":"2024-03-04T16:04:07.763022+0000","last_clean":"2024-03-04T16:04:07.763022+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:04:07.763022+0000","last_undegraded":"2024-03-04T16:04:07.763022+0000","last_fullsized":"2024-03-04T16:04:07.763022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.762953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:37.004167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.533334+0000","last_change":"2024-03-04T16:04:09.533365+0000","last_active":"2024-03-04T16:04:09.533334+0000","last_peered":"2024-03-04T16:04:09.533334+0000","last_clean":"2024-03-04T16:04:09.533334+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:04:09.533334+0000","last_undegraded":"2024-03-04T16:04:09.533334+0000","last_fullsized":"2024-03-04T16:04:09.533334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:04:09.533267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.533267+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.228405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.509705+0000","last_change":"2024-03-04T16:04:04.509748+0000","last_active":"2024-03-04T16:04:04.509705+0000","last_peered":"2024-03-04T16:04:04.509705+0000","last_clean":"2024-03-04T16:04:04.509705+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:04:04.509705+0000","last_undegraded":"2024-03-04T16:04:04.509705+0000","last_fullsized":"2024-03-04T16:04:04.509705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.509608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.509608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.020538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.774760+0000","last_change":"2024-03-04T16:02:47.770597+0000","last_active":"2024-03-04T16:04:03.774760+0000","last_peered":"2024-03-04T16:04:03.774760+0000","last_clean":"2024-03-04T16:04:03.774760+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:04:03.774760+0000","last_undegraded":"2024-03-04T16:04:03.774760+0000","last_fullsized":"2024-03-04T16:04:03.774760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096393+0000","last_change":"2024-03-04T16:02:43.715105+0000","last_active":"2024-03-04T16:04:00.096393+0000","last_peered":"2024-03-04T16:04:00.096393+0000","last_clean":"2024-03-04T16:04:00.096393+0000","last_became_active":"2024-03-04T16:02:43.714792+0000","last_became_peered":"2024-03-04T16:02:43.714792+0000","last_unstale":"2024-03-04T16:04:00.096393+0000","last_undegraded":"2024-03-04T16:04:00.096393+0000","last_fullsized":"2024-03-04T16:04:00.096393+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171318+0000","last_change":"2024-03-04T16:02:45.744111+0000","last_active":"2024-03-04T16:04:00.171318+0000","last_peered":"2024-03-04T16:04:00.171318+0000","last_clean":"2024-03-04T16:04:00.171318+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:04:00.171318+0000","last_undegraded":"2024-03-04T16:04:00.171318+0000","last_fullsized":"2024-03-04T16:04:00.171318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.765686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":411,"ondisk_log_size":411,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070486,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":946996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1388544,"data_stored":944684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":856064,"data_stored":356088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1413120,"data_stored":946340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":790528,"data_stored":354816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":356082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":355306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626464,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":354642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:04:17.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:04:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:04:17.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:16 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:04:17.295 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:17.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:04:17.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.b deep-scrub starts 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.b deep-scrub ok 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: from='client.14787 v1:172.21.15.38:0/104933251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:04:17.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.b deep-scrub starts 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.b deep-scrub ok 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: from='client.14787 v1:172.21.15.38:0/104933251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:04:17.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:17 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:04:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:04:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.b deep-scrub starts 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.b deep-scrub ok 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: from='client.14787 v1:172.21.15.38:0/104933251' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:04:17.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:17 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:04:18.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 4.7 deep-scrub starts 2024-03-04T16:04:18.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 4.7 deep-scrub ok 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 6.1c deep-scrub starts 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 6.1c deep-scrub ok 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 4.7 deep-scrub starts 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 4.7 deep-scrub ok 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 6.1c deep-scrub starts 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 6.1c deep-scrub ok 2024-03-04T16:04:18.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:04:18.720 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:18 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:04:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 4.7 deep-scrub starts 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 4.7 deep-scrub ok 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 6.1c deep-scrub starts 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 6.1c deep-scrub ok 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:04:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:18 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:04:19.325 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:19.327 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:04:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 5.d deep-scrub starts 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 5.d deep-scrub ok 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:04:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:19 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:04:19.859 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:19 smithi176 systemd[1]: Starting Ceph grafana.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:04:19.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 5.d deep-scrub starts 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 5.d deep-scrub ok 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:19.878 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 5.d deep-scrub starts 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 5.d deep-scrub ok 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: osdmap e91: 8 total, 8 up, 8 in 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:04:19.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:19 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:04:19.880 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-03-04T16:04:19.294152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1022,"num_read_kb":868,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":625,"ondisk_log_size":625,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240892,"kb_used_data":8124,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720324,"statfs":{"total":767960285184,"available":767713611776,"internally_reserved":0,"allocated":8318976,"data_stored":4614954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002443"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171404+0000","last_change":"2024-03-04T16:03:57.477127+0000","last_active":"2024-03-04T16:04:00.171404+0000","last_peered":"2024-03-04T16:04:00.171404+0000","last_clean":"2024-03-04T16:04:00.171404+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:04:00.171404+0000","last_undegraded":"2024-03-04T16:04:00.171404+0000","last_fullsized":"2024-03-04T16:04:00.171404+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.477014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.477014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.118199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170834+0000","last_change":"2024-03-04T16:02:47.767863+0000","last_active":"2024-03-04T16:04:00.170834+0000","last_peered":"2024-03-04T16:04:00.170834+0000","last_clean":"2024-03-04T16:04:00.170834+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:04:00.170834+0000","last_undegraded":"2024-03-04T16:04:00.170834+0000","last_fullsized":"2024-03-04T16:04:00.170834+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:04.395983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171558+0000","last_change":"2024-03-04T16:03:45.773330+0000","last_active":"2024-03-04T16:04:00.171558+0000","last_peered":"2024-03-04T16:04:00.171558+0000","last_clean":"2024-03-04T16:04:00.171558+0000","last_became_active":"2024-03-04T16:02:43.714846+0000","last_became_peered":"2024-03-04T16:02:43.714846+0000","last_unstale":"2024-03-04T16:04:00.171558+0000","last_undegraded":"2024-03-04T16:04:00.171558+0000","last_fullsized":"2024-03-04T16:04:00.171558+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:45.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.773213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.488104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:15.666735+0000","last_change":"2024-03-04T16:04:15.666765+0000","last_active":"2024-03-04T16:04:15.666735+0000","last_peered":"2024-03-04T16:04:15.666735+0000","last_clean":"2024-03-04T16:04:15.666735+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:04:15.666735+0000","last_undegraded":"2024-03-04T16:04:15.666735+0000","last_fullsized":"2024-03-04T16:04:15.666735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:15.666674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.666674+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.309413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.380440+0000","last_change":"2024-03-04T16:04:05.380482+0000","last_active":"2024-03-04T16:04:05.380440+0000","last_peered":"2024-03-04T16:04:05.380440+0000","last_clean":"2024-03-04T16:04:05.380440+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:04:05.380440+0000","last_undegraded":"2024-03-04T16:04:05.380440+0000","last_fullsized":"2024-03-04T16:04:05.380440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.380364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.380364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.798710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.519799+0000","last_change":"2024-03-04T16:04:08.519836+0000","last_active":"2024-03-04T16:04:08.519799+0000","last_peered":"2024-03-04T16:04:08.519799+0000","last_clean":"2024-03-04T16:04:08.519799+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:04:08.519799+0000","last_undegraded":"2024-03-04T16:04:08.519799+0000","last_fullsized":"2024-03-04T16:04:08.519799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.519713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.519713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.148055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096811+0000","last_change":"2024-03-04T16:02:43.712774+0000","last_active":"2024-03-04T16:04:00.096811+0000","last_peered":"2024-03-04T16:04:00.096811+0000","last_clean":"2024-03-04T16:04:00.096811+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:04:00.096811+0000","last_undegraded":"2024-03-04T16:04:00.096811+0000","last_fullsized":"2024-03-04T16:04:00.096811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.121882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096197+0000","last_change":"2024-03-04T16:03:50.379226+0000","last_active":"2024-03-04T16:04:00.096197+0000","last_peered":"2024-03-04T16:04:00.096197+0000","last_clean":"2024-03-04T16:04:00.096197+0000","last_became_active":"2024-03-04T16:02:45.749383+0000","last_became_peered":"2024-03-04T16:02:45.749383+0000","last_unstale":"2024-03-04T16:04:00.096197+0000","last_undegraded":"2024-03-04T16:04:00.096197+0000","last_fullsized":"2024-03-04T16:04:00.096197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:50.379077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.379077+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.153393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170489+0000","last_change":"2024-03-04T16:03:56.851189+0000","last_active":"2024-03-04T16:04:00.170489+0000","last_peered":"2024-03-04T16:04:00.170489+0000","last_clean":"2024-03-04T16:04:00.170489+0000","last_became_active":"2024-03-04T16:02:49.791142+0000","last_became_peered":"2024-03-04T16:02:49.791142+0000","last_unstale":"2024-03-04T16:04:00.170489+0000","last_undegraded":"2024-03-04T16:04:00.170489+0000","last_fullsized":"2024-03-04T16:04:00.170489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.851094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.851094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.117129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171309+0000","last_change":"2024-03-04T16:02:47.775367+0000","last_active":"2024-03-04T16:04:00.171309+0000","last_peered":"2024-03-04T16:04:00.171309+0000","last_clean":"2024-03-04T16:04:00.171309+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:04:00.171309+0000","last_undegraded":"2024-03-04T16:04:00.171309+0000","last_fullsized":"2024-03-04T16:04:00.171309+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.508809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172396+0000","last_change":"2024-03-04T16:03:51.290213+0000","last_active":"2024-03-04T16:04:00.172396+0000","last_peered":"2024-03-04T16:04:00.172396+0000","last_clean":"2024-03-04T16:04:00.172396+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:04:00.172396+0000","last_undegraded":"2024-03-04T16:04:00.172396+0000","last_fullsized":"2024-03-04T16:04:00.172396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.290117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.290117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.885059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.434104+0000","last_change":"2024-03-04T16:04:00.434148+0000","last_active":"2024-03-04T16:04:00.434104+0000","last_peered":"2024-03-04T16:04:00.434104+0000","last_clean":"2024-03-04T16:04:00.434104+0000","last_became_active":"2024-03-04T16:02:45.745532+0000","last_became_peered":"2024-03-04T16:02:45.745532+0000","last_unstale":"2024-03-04T16:04:00.434104+0000","last_undegraded":"2024-03-04T16:04:00.434104+0000","last_fullsized":"2024-03-04T16:04:00.434104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.434035+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.035919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:02.311200+0000","last_change":"2024-03-04T16:04:02.311228+0000","last_active":"2024-03-04T16:04:02.311200+0000","last_peered":"2024-03-04T16:04:02.311200+0000","last_clean":"2024-03-04T16:04:02.311200+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:04:02.311200+0000","last_undegraded":"2024-03-04T16:04:02.311200+0000","last_fullsized":"2024-03-04T16:04:02.311200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:02.311103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:02.311103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:06.585200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.905847+0000","last_change":"2024-03-04T16:04:08.905885+0000","last_active":"2024-03-04T16:04:08.905847+0000","last_peered":"2024-03-04T16:04:08.905847+0000","last_clean":"2024-03-04T16:04:08.905847+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:04:08.905847+0000","last_undegraded":"2024-03-04T16:04:08.905847+0000","last_fullsized":"2024-03-04T16:04:08.905847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.905768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.905768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.441912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.292580+0000","last_change":"2024-03-04T16:04:00.292627+0000","last_active":"2024-03-04T16:04:00.292580+0000","last_peered":"2024-03-04T16:04:00.292580+0000","last_clean":"2024-03-04T16:04:00.292580+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:04:00.292580+0000","last_undegraded":"2024-03-04T16:04:00.292580+0000","last_fullsized":"2024-03-04T16:04:00.292580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:00.292499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.292499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.161517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171851+0000","last_change":"2024-03-04T16:02:45.748750+0000","last_active":"2024-03-04T16:04:00.171851+0000","last_peered":"2024-03-04T16:04:00.171851+0000","last_clean":"2024-03-04T16:04:00.171851+0000","last_became_active":"2024-03-04T16:02:45.747787+0000","last_became_peered":"2024-03-04T16:02:45.747787+0000","last_unstale":"2024-03-04T16:04:00.171851+0000","last_undegraded":"2024-03-04T16:04:00.171851+0000","last_fullsized":"2024-03-04T16:04:00.171851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:14.663457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172933+0000","last_change":"2024-03-04T16:02:49.785706+0000","last_active":"2024-03-04T16:04:00.172933+0000","last_peered":"2024-03-04T16:04:00.172933+0000","last_clean":"2024-03-04T16:04:00.172933+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:04:00.172933+0000","last_undegraded":"2024-03-04T16:04:00.172933+0000","last_fullsized":"2024-03-04T16:04:00.172933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.708627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.039716+0000","last_change":"2024-03-04T16:04:08.039737+0000","last_active":"2024-03-04T16:04:08.039716+0000","last_peered":"2024-03-04T16:04:08.039716+0000","last_clean":"2024-03-04T16:04:08.039716+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:04:08.039716+0000","last_undegraded":"2024-03-04T16:04:08.039716+0000","last_fullsized":"2024-03-04T16:04:08.039716+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.039649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.039649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.929286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:12.724420+0000","last_change":"2024-03-04T16:04:12.724442+0000","last_active":"2024-03-04T16:04:12.724420+0000","last_peered":"2024-03-04T16:04:12.724420+0000","last_clean":"2024-03-04T16:04:12.724420+0000","last_became_active":"2024-03-04T16:02:43.714419+0000","last_became_peered":"2024-03-04T16:02:43.714419+0000","last_unstale":"2024-03-04T16:04:12.724420+0000","last_undegraded":"2024-03-04T16:04:12.724420+0000","last_fullsized":"2024-03-04T16:04:12.724420+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:12.724360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:12.724360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:18.288024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095293+0000","last_change":"2024-03-04T16:03:45.980725+0000","last_active":"2024-03-04T16:04:00.095293+0000","last_peered":"2024-03-04T16:04:00.095293+0000","last_clean":"2024-03-04T16:04:00.095293+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:04:00.095293+0000","last_undegraded":"2024-03-04T16:04:00.095293+0000","last_fullsized":"2024-03-04T16:04:00.095293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:45.980606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.980606+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:01.702710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096882+0000","last_change":"2024-03-04T16:02:49.794035+0000","last_active":"2024-03-04T16:04:00.096882+0000","last_peered":"2024-03-04T16:04:00.096882+0000","last_clean":"2024-03-04T16:04:00.096882+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:04:00.096882+0000","last_undegraded":"2024-03-04T16:04:00.096882+0000","last_fullsized":"2024-03-04T16:04:00.096882+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:01.379614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.650756+0000","last_change":"2024-03-04T16:04:16.650776+0000","last_active":"2024-03-04T16:04:16.650756+0000","last_peered":"2024-03-04T16:04:16.650756+0000","last_clean":"2024-03-04T16:04:16.650756+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:04:16.650756+0000","last_undegraded":"2024-03-04T16:04:16.650756+0000","last_fullsized":"2024-03-04T16:04:16.650756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.650697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.650697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.898928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171760+0000","last_change":"2024-03-04T16:03:46.423866+0000","last_active":"2024-03-04T16:04:00.171760+0000","last_peered":"2024-03-04T16:04:00.171760+0000","last_clean":"2024-03-04T16:04:00.171760+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:04:00.171760+0000","last_undegraded":"2024-03-04T16:04:00.171760+0000","last_fullsized":"2024-03-04T16:04:00.171760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.423767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.868075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096446+0000","last_change":"2024-03-04T16:03:55.411658+0000","last_active":"2024-03-04T16:04:00.096446+0000","last_peered":"2024-03-04T16:04:00.096446+0000","last_clean":"2024-03-04T16:04:00.096446+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:04:00.096446+0000","last_undegraded":"2024-03-04T16:04:00.096446+0000","last_fullsized":"2024-03-04T16:04:00.096446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:55.411554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.411554+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.742258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171968+0000","last_change":"2024-03-04T16:02:49.792726+0000","last_active":"2024-03-04T16:04:00.171968+0000","last_peered":"2024-03-04T16:04:00.171968+0000","last_clean":"2024-03-04T16:04:00.171968+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:04:00.171968+0000","last_undegraded":"2024-03-04T16:04:00.171968+0000","last_fullsized":"2024-03-04T16:04:00.171968+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:06.657727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.707358+0000","last_change":"2024-03-04T16:04:09.707358+0000","last_active":"2024-03-04T16:04:09.707358+0000","last_peered":"2024-03-04T16:04:09.707358+0000","last_clean":"2024-03-04T16:04:09.707358+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:04:09.707358+0000","last_undegraded":"2024-03-04T16:04:09.707358+0000","last_fullsized":"2024-03-04T16:04:09.707358+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.707234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.707234+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.027920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:01.014974+0000","last_change":"2024-03-04T16:04:01.015009+0000","last_active":"2024-03-04T16:04:01.014974+0000","last_peered":"2024-03-04T16:04:01.014974+0000","last_clean":"2024-03-04T16:04:01.014974+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:04:01.014974+0000","last_undegraded":"2024-03-04T16:04:01.014974+0000","last_fullsized":"2024-03-04T16:04:01.014974+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:01.014871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:01.014871+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:13.772804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171781+0000","last_change":"2024-03-04T16:02:45.745063+0000","last_active":"2024-03-04T16:04:00.171781+0000","last_peered":"2024-03-04T16:04:00.171781+0000","last_clean":"2024-03-04T16:04:00.171781+0000","last_became_active":"2024-03-04T16:02:45.744705+0000","last_became_peered":"2024-03-04T16:02:45.744705+0000","last_unstale":"2024-03-04T16:04:00.171781+0000","last_undegraded":"2024-03-04T16:04:00.171781+0000","last_fullsized":"2024-03-04T16:04:00.171781+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.665716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171960+0000","last_change":"2024-03-04T16:03:53.955279+0000","last_active":"2024-03-04T16:04:00.171960+0000","last_peered":"2024-03-04T16:04:00.171960+0000","last_clean":"2024-03-04T16:04:00.171960+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:04:00.171960+0000","last_undegraded":"2024-03-04T16:04:00.171960+0000","last_fullsized":"2024-03-04T16:04:00.171960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:53.955186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:53.955186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.764558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171905+0000","last_change":"2024-03-04T16:03:57.874266+0000","last_active":"2024-03-04T16:04:00.171905+0000","last_peered":"2024-03-04T16:04:00.171905+0000","last_clean":"2024-03-04T16:04:00.171905+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:04:00.171905+0000","last_undegraded":"2024-03-04T16:04:00.171905+0000","last_fullsized":"2024-03-04T16:04:00.171905+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.874150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.874150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.705755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095851+0000","last_change":"2024-03-04T16:03:47.354437+0000","last_active":"2024-03-04T16:04:00.095851+0000","last_peered":"2024-03-04T16:04:00.095851+0000","last_clean":"2024-03-04T16:04:00.095851+0000","last_became_active":"2024-03-04T16:02:43.718570+0000","last_became_peered":"2024-03-04T16:02:43.718570+0000","last_unstale":"2024-03-04T16:04:00.095851+0000","last_undegraded":"2024-03-04T16:04:00.095851+0000","last_fullsized":"2024-03-04T16:04:00.095851+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.354324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.354324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.540186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087619+0000","last_change":"2024-03-04T16:03:51.720621+0000","last_active":"2024-03-04T16:04:00.087619+0000","last_peered":"2024-03-04T16:04:00.087619+0000","last_clean":"2024-03-04T16:04:00.087619+0000","last_became_active":"2024-03-04T16:02:45.746864+0000","last_became_peered":"2024-03-04T16:02:45.746864+0000","last_unstale":"2024-03-04T16:04:00.087619+0000","last_undegraded":"2024-03-04T16:04:00.087619+0000","last_fullsized":"2024-03-04T16:04:00.087619+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:51.720518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.720518+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.935701+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171107+0000","last_change":"2024-03-04T16:03:57.893125+0000","last_active":"2024-03-04T16:04:00.171107+0000","last_peered":"2024-03-04T16:04:00.171107+0000","last_clean":"2024-03-04T16:04:00.171107+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:04:00.171107+0000","last_undegraded":"2024-03-04T16:04:00.171107+0000","last_fullsized":"2024-03-04T16:04:00.171107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.893007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.893007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.709209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.798890+0000","last_change":"2024-03-04T16:04:05.798912+0000","last_active":"2024-03-04T16:04:05.798890+0000","last_peered":"2024-03-04T16:04:05.798890+0000","last_clean":"2024-03-04T16:04:05.798890+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:04:05.798890+0000","last_undegraded":"2024-03-04T16:04:05.798890+0000","last_fullsized":"2024-03-04T16:04:05.798890+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.798829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.798829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.585226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.761398+0000","last_change":"2024-03-04T16:04:08.761418+0000","last_active":"2024-03-04T16:04:08.761398+0000","last_peered":"2024-03-04T16:04:08.761398+0000","last_clean":"2024-03-04T16:04:08.761398+0000","last_became_active":"2024-03-04T16:02:43.716542+0000","last_became_peered":"2024-03-04T16:02:43.716542+0000","last_unstale":"2024-03-04T16:04:08.761398+0000","last_undegraded":"2024-03-04T16:04:08.761398+0000","last_fullsized":"2024-03-04T16:04:08.761398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.761341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.761341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:10.153431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095860+0000","last_change":"2024-03-04T16:02:45.754181+0000","last_active":"2024-03-04T16:04:00.095860+0000","last_peered":"2024-03-04T16:04:00.095860+0000","last_clean":"2024-03-04T16:04:00.095860+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:04:00.095860+0000","last_undegraded":"2024-03-04T16:04:00.095860+0000","last_fullsized":"2024-03-04T16:04:00.095860+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:00.606250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096015+0000","last_change":"2024-03-04T16:02:49.791755+0000","last_active":"2024-03-04T16:04:00.096015+0000","last_peered":"2024-03-04T16:04:00.096015+0000","last_clean":"2024-03-04T16:04:00.096015+0000","last_became_active":"2024-03-04T16:02:49.791216+0000","last_became_peered":"2024-03-04T16:02:49.791216+0000","last_unstale":"2024-03-04T16:04:00.096015+0000","last_undegraded":"2024-03-04T16:04:00.096015+0000","last_fullsized":"2024-03-04T16:04:00.096015+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.261130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171216+0000","last_change":"2024-03-04T16:02:47.773727+0000","last_active":"2024-03-04T16:04:00.171216+0000","last_peered":"2024-03-04T16:04:00.171216+0000","last_clean":"2024-03-04T16:04:00.171216+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:04:00.171216+0000","last_undegraded":"2024-03-04T16:04:00.171216+0000","last_fullsized":"2024-03-04T16:04:00.171216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.258206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171414+0000","last_change":"2024-03-04T16:03:52.330168+0000","last_active":"2024-03-04T16:04:00.171414+0000","last_peered":"2024-03-04T16:04:00.171414+0000","last_clean":"2024-03-04T16:04:00.171414+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:04:00.171414+0000","last_undegraded":"2024-03-04T16:04:00.171414+0000","last_fullsized":"2024-03-04T16:04:00.171414+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.330026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.330026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.392092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:13.431813+0000","last_change":"2024-03-04T16:04:13.431846+0000","last_active":"2024-03-04T16:04:13.431813+0000","last_peered":"2024-03-04T16:04:13.431813+0000","last_clean":"2024-03-04T16:04:13.431813+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:04:13.431813+0000","last_undegraded":"2024-03-04T16:04:13.431813+0000","last_fullsized":"2024-03-04T16:04:13.431813+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:04:13.431752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.431752+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.426748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.790980+0000","last_change":"2024-03-04T16:04:03.791013+0000","last_active":"2024-03-04T16:04:03.790980+0000","last_peered":"2024-03-04T16:04:03.790980+0000","last_clean":"2024-03-04T16:04:03.790980+0000","last_became_active":"2024-03-04T16:02:49.794197+0000","last_became_peered":"2024-03-04T16:02:49.794197+0000","last_unstale":"2024-03-04T16:04:03.790980+0000","last_undegraded":"2024-03-04T16:04:03.790980+0000","last_fullsized":"2024-03-04T16:04:03.790980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.790905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.790905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.304942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096301+0000","last_change":"2024-03-04T16:02:47.777366+0000","last_active":"2024-03-04T16:04:00.096301+0000","last_peered":"2024-03-04T16:04:00.096301+0000","last_clean":"2024-03-04T16:04:00.096301+0000","last_became_active":"2024-03-04T16:02:47.777008+0000","last_became_peered":"2024-03-04T16:02:47.777008+0000","last_unstale":"2024-03-04T16:04:00.096301+0000","last_undegraded":"2024-03-04T16:04:00.096301+0000","last_fullsized":"2024-03-04T16:04:00.096301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.335199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172131+0000","last_change":"2024-03-04T16:02:43.711547+0000","last_active":"2024-03-04T16:04:00.172131+0000","last_peered":"2024-03-04T16:04:00.172131+0000","last_clean":"2024-03-04T16:04:00.172131+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:04:00.172131+0000","last_undegraded":"2024-03-04T16:04:00.172131+0000","last_fullsized":"2024-03-04T16:04:00.172131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.527444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":79,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.040200+0000","last_change":"2024-03-04T16:04:07.040237+0000","last_active":"2024-03-04T16:04:07.040200+0000","last_peered":"2024-03-04T16:04:07.040200+0000","last_clean":"2024-03-04T16:04:07.040200+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:04:07.040200+0000","last_undegraded":"2024-03-04T16:04:07.040200+0000","last_fullsized":"2024-03-04T16:04:07.040200+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.040136+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.386047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095861+0000","last_change":"2024-03-04T16:02:49.792155+0000","last_active":"2024-03-04T16:04:00.095861+0000","last_peered":"2024-03-04T16:04:00.095861+0000","last_clean":"2024-03-04T16:04:00.095861+0000","last_became_active":"2024-03-04T16:02:49.791815+0000","last_became_peered":"2024-03-04T16:02:49.791815+0000","last_unstale":"2024-03-04T16:04:00.095861+0000","last_undegraded":"2024-03-04T16:04:00.095861+0000","last_fullsized":"2024-03-04T16:04:00.095861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.727768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170892+0000","last_change":"2024-03-04T16:03:51.939795+0000","last_active":"2024-03-04T16:04:00.170892+0000","last_peered":"2024-03-04T16:04:00.170892+0000","last_clean":"2024-03-04T16:04:00.170892+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:04:00.170892+0000","last_undegraded":"2024-03-04T16:04:00.170892+0000","last_fullsized":"2024-03-04T16:04:00.170892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.939712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.939712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.441874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171382+0000","last_change":"2024-03-04T16:03:44.419164+0000","last_active":"2024-03-04T16:04:00.171382+0000","last_peered":"2024-03-04T16:04:00.171382+0000","last_clean":"2024-03-04T16:04:00.171382+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:04:00.171382+0000","last_undegraded":"2024-03-04T16:04:00.171382+0000","last_fullsized":"2024-03-04T16:04:00.171382+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:44.419065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:44.419065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.867222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171695+0000","last_change":"2024-03-04T16:03:50.400887+0000","last_active":"2024-03-04T16:04:00.171695+0000","last_peered":"2024-03-04T16:04:00.171695+0000","last_clean":"2024-03-04T16:04:00.171695+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:04:00.171695+0000","last_undegraded":"2024-03-04T16:04:00.171695+0000","last_fullsized":"2024-03-04T16:04:00.171695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:50.400784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.400784+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.749407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171573+0000","last_change":"2024-03-04T16:02:49.797503+0000","last_active":"2024-03-04T16:04:00.171573+0000","last_peered":"2024-03-04T16:04:00.171573+0000","last_clean":"2024-03-04T16:04:00.171573+0000","last_became_active":"2024-03-04T16:02:49.797174+0000","last_became_peered":"2024-03-04T16:02:49.797174+0000","last_unstale":"2024-03-04T16:04:00.171573+0000","last_undegraded":"2024-03-04T16:04:00.171573+0000","last_fullsized":"2024-03-04T16:04:00.171573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:17.229451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.217159+0000","last_change":"2024-03-04T16:04:10.217159+0000","last_active":"2024-03-04T16:04:10.217159+0000","last_peered":"2024-03-04T16:04:10.217159+0000","last_clean":"2024-03-04T16:04:10.217159+0000","last_became_active":"2024-03-04T16:02:47.768874+0000","last_became_peered":"2024-03-04T16:02:47.768874+0000","last_unstale":"2024-03-04T16:04:10.217159+0000","last_undegraded":"2024-03-04T16:04:10.217159+0000","last_fullsized":"2024-03-04T16:04:10.217159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:10.217024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.217024+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:39.681485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.837345+0000","last_change":"2024-03-04T16:04:06.837374+0000","last_active":"2024-03-04T16:04:06.837345+0000","last_peered":"2024-03-04T16:04:06.837345+0000","last_clean":"2024-03-04T16:04:06.837345+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:04:06.837345+0000","last_undegraded":"2024-03-04T16:04:06.837345+0000","last_fullsized":"2024-03-04T16:04:06.837345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:06.837263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.837263+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.977815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096389+0000","last_change":"2024-03-04T16:02:45.749382+0000","last_active":"2024-03-04T16:04:00.096389+0000","last_peered":"2024-03-04T16:04:00.096389+0000","last_clean":"2024-03-04T16:04:00.096389+0000","last_became_active":"2024-03-04T16:02:45.748824+0000","last_became_peered":"2024-03-04T16:02:45.748824+0000","last_unstale":"2024-03-04T16:04:00.096389+0000","last_undegraded":"2024-03-04T16:04:00.096389+0000","last_fullsized":"2024-03-04T16:04:00.096389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:05.070717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.580370+0000","last_change":"2024-03-04T16:04:10.580391+0000","last_active":"2024-03-04T16:04:10.580370+0000","last_peered":"2024-03-04T16:04:10.580370+0000","last_clean":"2024-03-04T16:04:10.580370+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:04:10.580370+0000","last_undegraded":"2024-03-04T16:04:10.580370+0000","last_fullsized":"2024-03-04T16:04:10.580370+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:10.580305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.580305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.834007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775916+0000","last_change":"2024-03-04T16:02:47.774221+0000","last_active":"2024-03-04T16:04:08.775916+0000","last_peered":"2024-03-04T16:04:08.775916+0000","last_clean":"2024-03-04T16:04:08.775916+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:04:08.775916+0000","last_undegraded":"2024-03-04T16:04:08.775916+0000","last_fullsized":"2024-03-04T16:04:08.775916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170631+0000","last_change":"2024-03-04T16:03:49.810121+0000","last_active":"2024-03-04T16:04:00.170631+0000","last_peered":"2024-03-04T16:04:00.170631+0000","last_clean":"2024-03-04T16:04:00.170631+0000","last_became_active":"2024-03-04T16:02:43.713331+0000","last_became_peered":"2024-03-04T16:02:43.713331+0000","last_unstale":"2024-03-04T16:04:00.170631+0000","last_undegraded":"2024-03-04T16:04:00.170631+0000","last_fullsized":"2024-03-04T16:04:00.170631+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:03:49.810002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:49.810002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.948952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172670+0000","last_change":"2024-03-04T16:03:55.306308+0000","last_active":"2024-03-04T16:04:00.172670+0000","last_peered":"2024-03-04T16:04:00.172670+0000","last_clean":"2024-03-04T16:04:00.172670+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:04:00.172670+0000","last_undegraded":"2024-03-04T16:04:00.172670+0000","last_fullsized":"2024-03-04T16:04:00.172670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:03:55.306180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.306180+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.830154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:13.176819+0000","last_change":"2024-03-04T16:04:13.176858+0000","last_active":"2024-03-04T16:04:13.176819+0000","last_peered":"2024-03-04T16:04:13.176819+0000","last_clean":"2024-03-04T16:04:13.176819+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:04:13.176819+0000","last_undegraded":"2024-03-04T16:04:13.176819+0000","last_fullsized":"2024-03-04T16:04:13.176819+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:13.176740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.176740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:28.778125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:14.387289+0000","last_change":"2024-03-04T16:04:14.387312+0000","last_active":"2024-03-04T16:04:14.387289+0000","last_peered":"2024-03-04T16:04:14.387289+0000","last_clean":"2024-03-04T16:04:14.387289+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:04:14.387289+0000","last_undegraded":"2024-03-04T16:04:14.387289+0000","last_fullsized":"2024-03-04T16:04:14.387289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:14.387217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.387217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.518011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171544+0000","last_change":"2024-03-04T16:02:43.714055+0000","last_active":"2024-03-04T16:04:00.171544+0000","last_peered":"2024-03-04T16:04:00.171544+0000","last_clean":"2024-03-04T16:04:00.171544+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:04:00.171544+0000","last_undegraded":"2024-03-04T16:04:00.171544+0000","last_fullsized":"2024-03-04T16:04:00.171544+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:08.115403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:05.755480+0000","last_change":"2024-03-04T16:04:05.755536+0000","last_active":"2024-03-04T16:04:05.755480+0000","last_peered":"2024-03-04T16:04:05.755480+0000","last_clean":"2024-03-04T16:04:05.755480+0000","last_became_active":"2024-03-04T16:02:45.745571+0000","last_became_peered":"2024-03-04T16:02:45.745571+0000","last_unstale":"2024-03-04T16:04:05.755480+0000","last_undegraded":"2024-03-04T16:04:05.755480+0000","last_fullsized":"2024-03-04T16:04:05.755480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:05.755370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.755370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.469566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.871295+0000","last_change":"2024-03-04T16:04:09.871332+0000","last_active":"2024-03-04T16:04:09.871295+0000","last_peered":"2024-03-04T16:04:09.871295+0000","last_clean":"2024-03-04T16:04:09.871295+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:04:09.871295+0000","last_undegraded":"2024-03-04T16:04:09.871295+0000","last_fullsized":"2024-03-04T16:04:09.871295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.871214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.871214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:30.539294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:11.175170+0000","last_change":"2024-03-04T16:04:11.175190+0000","last_active":"2024-03-04T16:04:11.175170+0000","last_peered":"2024-03-04T16:04:11.175170+0000","last_clean":"2024-03-04T16:04:11.175170+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:04:11.175170+0000","last_undegraded":"2024-03-04T16:04:11.175170+0000","last_fullsized":"2024-03-04T16:04:11.175170+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:11.175113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:11.175113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.876575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.086907+0000","last_change":"2024-03-04T16:03:46.718298+0000","last_active":"2024-03-04T16:04:00.086907+0000","last_peered":"2024-03-04T16:04:00.086907+0000","last_clean":"2024-03-04T16:04:00.086907+0000","last_became_active":"2024-03-04T16:02:43.718241+0000","last_became_peered":"2024-03-04T16:02:43.718241+0000","last_unstale":"2024-03-04T16:04:00.086907+0000","last_undegraded":"2024-03-04T16:04:00.086907+0000","last_fullsized":"2024-03-04T16:04:00.086907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.718188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.606455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096922+0000","last_change":"2024-03-04T16:03:52.421377+0000","last_active":"2024-03-04T16:04:00.096922+0000","last_peered":"2024-03-04T16:04:00.096922+0000","last_clean":"2024-03-04T16:04:00.096922+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:04:00.096922+0000","last_undegraded":"2024-03-04T16:04:00.096922+0000","last_fullsized":"2024-03-04T16:04:00.096922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.421285+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.703150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171188+0000","last_change":"2024-03-04T16:03:58.313071+0000","last_active":"2024-03-04T16:04:00.171188+0000","last_peered":"2024-03-04T16:04:00.171188+0000","last_clean":"2024-03-04T16:04:00.171188+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:04:00.171188+0000","last_undegraded":"2024-03-04T16:04:00.171188+0000","last_fullsized":"2024-03-04T16:04:00.171188+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.312925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.312925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.584207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.869070+0000","last_change":"2024-03-04T16:04:16.869117+0000","last_active":"2024-03-04T16:04:16.869070+0000","last_peered":"2024-03-04T16:04:16.869070+0000","last_clean":"2024-03-04T16:04:16.869070+0000","last_became_active":"2024-03-04T16:02:43.716945+0000","last_became_peered":"2024-03-04T16:02:43.716945+0000","last_unstale":"2024-03-04T16:04:16.869070+0000","last_undegraded":"2024-03-04T16:04:16.869070+0000","last_fullsized":"2024-03-04T16:04:16.869070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:04:16.868993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.868993+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.502678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170668+0000","last_change":"2024-03-04T16:03:48.260185+0000","last_active":"2024-03-04T16:04:00.170668+0000","last_peered":"2024-03-04T16:04:00.170668+0000","last_clean":"2024-03-04T16:04:00.170668+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:04:00.170668+0000","last_undegraded":"2024-03-04T16:04:00.170668+0000","last_fullsized":"2024-03-04T16:04:00.170668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:48.260047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.260047+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.815204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775727+0000","last_change":"2024-03-04T16:02:47.774009+0000","last_active":"2024-03-04T16:04:08.775727+0000","last_peered":"2024-03-04T16:04:08.775727+0000","last_clean":"2024-03-04T16:04:08.775727+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:04:08.775727+0000","last_undegraded":"2024-03-04T16:04:08.775727+0000","last_fullsized":"2024-03-04T16:04:08.775727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096118+0000","last_change":"2024-03-04T16:02:49.796054+0000","last_active":"2024-03-04T16:04:00.096118+0000","last_peered":"2024-03-04T16:04:00.096118+0000","last_clean":"2024-03-04T16:04:00.096118+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:04:00.096118+0000","last_undegraded":"2024-03-04T16:04:00.096118+0000","last_fullsized":"2024-03-04T16:04:00.096118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.821795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172343+0000","last_change":"2024-03-04T16:03:55.817828+0000","last_active":"2024-03-04T16:04:00.172343+0000","last_peered":"2024-03-04T16:04:00.172343+0000","last_clean":"2024-03-04T16:04:00.172343+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:04:00.172343+0000","last_undegraded":"2024-03-04T16:04:00.172343+0000","last_fullsized":"2024-03-04T16:04:00.172343+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:55.817709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.817709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.939046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171220+0000","last_change":"2024-03-04T16:03:48.407212+0000","last_active":"2024-03-04T16:04:00.171220+0000","last_peered":"2024-03-04T16:04:00.171220+0000","last_clean":"2024-03-04T16:04:00.171220+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:04:00.171220+0000","last_undegraded":"2024-03-04T16:04:00.171220+0000","last_fullsized":"2024-03-04T16:04:00.171220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:48.407088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.407088+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:49.238194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095123+0000","last_change":"2024-03-04T16:03:59.048702+0000","last_active":"2024-03-04T16:04:00.095123+0000","last_peered":"2024-03-04T16:04:00.095123+0000","last_clean":"2024-03-04T16:04:00.095123+0000","last_became_active":"2024-03-04T16:02:47.767316+0000","last_became_peered":"2024-03-04T16:02:47.767316+0000","last_unstale":"2024-03-04T16:04:00.095123+0000","last_undegraded":"2024-03-04T16:04:00.095123+0000","last_fullsized":"2024-03-04T16:04:00.095123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:04.396312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170696+0000","last_change":"2024-03-04T16:02:49.797192+0000","last_active":"2024-03-04T16:04:00.170696+0000","last_peered":"2024-03-04T16:04:00.170696+0000","last_clean":"2024-03-04T16:04:00.170696+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:04:00.170696+0000","last_undegraded":"2024-03-04T16:04:00.170696+0000","last_fullsized":"2024-03-04T16:04:00.170696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.448969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095320+0000","last_change":"2024-03-04T16:03:51.984808+0000","last_active":"2024-03-04T16:04:00.095320+0000","last_peered":"2024-03-04T16:04:00.095320+0000","last_clean":"2024-03-04T16:04:00.095320+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:04:00.095320+0000","last_undegraded":"2024-03-04T16:04:00.095320+0000","last_fullsized":"2024-03-04T16:04:00.095320+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.984721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.984721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.092310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172295+0000","last_change":"2024-03-04T16:02:43.715496+0000","last_active":"2024-03-04T16:04:00.172295+0000","last_peered":"2024-03-04T16:04:00.172295+0000","last_clean":"2024-03-04T16:04:00.172295+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:04:00.172295+0000","last_undegraded":"2024-03-04T16:04:00.172295+0000","last_fullsized":"2024-03-04T16:04:00.172295+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.926799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":90,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:13.665469+0000","last_change":"2024-03-04T16:04:13.665506+0000","last_active":"2024-03-04T16:04:13.665469+0000","last_peered":"2024-03-04T16:04:13.665469+0000","last_clean":"2024-03-04T16:04:13.665469+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:04:13.665469+0000","last_undegraded":"2024-03-04T16:04:13.665469+0000","last_fullsized":"2024-03-04T16:04:13.665469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:04:13.665406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.665406+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.760464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170394+0000","last_change":"2024-03-04T16:03:50.786884+0000","last_active":"2024-03-04T16:04:00.170394+0000","last_peered":"2024-03-04T16:04:00.170394+0000","last_clean":"2024-03-04T16:04:00.170394+0000","last_became_active":"2024-03-04T16:02:49.793023+0000","last_became_peered":"2024-03-04T16:02:49.793023+0000","last_unstale":"2024-03-04T16:04:00.170394+0000","last_undegraded":"2024-03-04T16:04:00.170394+0000","last_fullsized":"2024-03-04T16:04:00.170394+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.786805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.786805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.570842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095255+0000","last_change":"2024-03-04T16:02:47.769558+0000","last_active":"2024-03-04T16:04:00.095255+0000","last_peered":"2024-03-04T16:04:00.095255+0000","last_clean":"2024-03-04T16:04:00.095255+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:04:00.095255+0000","last_undegraded":"2024-03-04T16:04:00.095255+0000","last_fullsized":"2024-03-04T16:04:00.095255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.736215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.230097+0000","last_change":"2024-03-04T16:04:06.230119+0000","last_active":"2024-03-04T16:04:06.230097+0000","last_peered":"2024-03-04T16:04:06.230097+0000","last_clean":"2024-03-04T16:04:06.230097+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:04:06.230097+0000","last_undegraded":"2024-03-04T16:04:06.230097+0000","last_fullsized":"2024-03-04T16:04:06.230097+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.230030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.804736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:14.777029+0000","last_change":"2024-03-04T16:04:14.777059+0000","last_active":"2024-03-04T16:04:14.777029+0000","last_peered":"2024-03-04T16:04:14.777029+0000","last_clean":"2024-03-04T16:04:14.777029+0000","last_became_active":"2024-03-04T16:02:45.752201+0000","last_became_peered":"2024-03-04T16:02:45.752201+0000","last_unstale":"2024-03-04T16:04:14.777029+0000","last_undegraded":"2024-03-04T16:04:14.777029+0000","last_fullsized":"2024-03-04T16:04:14.777029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:14.776970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.776970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.521202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096551+0000","last_change":"2024-03-04T16:02:49.793485+0000","last_active":"2024-03-04T16:04:00.096551+0000","last_peered":"2024-03-04T16:04:00.096551+0000","last_clean":"2024-03-04T16:04:00.096551+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:04:00.096551+0000","last_undegraded":"2024-03-04T16:04:00.096551+0000","last_fullsized":"2024-03-04T16:04:00.096551+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.099839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170930+0000","last_change":"2024-03-04T16:03:54.344325+0000","last_active":"2024-03-04T16:04:00.170930+0000","last_peered":"2024-03-04T16:04:00.170930+0000","last_clean":"2024-03-04T16:04:00.170930+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:04:00.170930+0000","last_undegraded":"2024-03-04T16:04:00.170930+0000","last_fullsized":"2024-03-04T16:04:00.170930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:03:54.344193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:54.344193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.913538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171536+0000","last_change":"2024-03-04T16:03:59.437627+0000","last_active":"2024-03-04T16:04:00.171536+0000","last_peered":"2024-03-04T16:04:00.171536+0000","last_clean":"2024-03-04T16:04:00.171536+0000","last_became_active":"2024-03-04T16:02:43.717129+0000","last_became_peered":"2024-03-04T16:02:43.717129+0000","last_unstale":"2024-03-04T16:04:00.171536+0000","last_undegraded":"2024-03-04T16:04:00.171536+0000","last_fullsized":"2024-03-04T16:04:00.171536+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.437524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.437524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.171391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775608+0000","last_change":"2024-03-04T16:02:45.753468+0000","last_active":"2024-03-04T16:04:08.775608+0000","last_peered":"2024-03-04T16:04:08.775608+0000","last_clean":"2024-03-04T16:04:08.775608+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:04:08.775608+0000","last_undegraded":"2024-03-04T16:04:08.775608+0000","last_fullsized":"2024-03-04T16:04:08.775608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171374+0000","last_change":"2024-03-04T16:02:47.773975+0000","last_active":"2024-03-04T16:04:00.171374+0000","last_peered":"2024-03-04T16:04:00.171374+0000","last_clean":"2024-03-04T16:04:00.171374+0000","last_became_active":"2024-03-04T16:02:47.773547+0000","last_became_peered":"2024-03-04T16:02:47.773547+0000","last_unstale":"2024-03-04T16:04:00.171374+0000","last_undegraded":"2024-03-04T16:04:00.171374+0000","last_fullsized":"2024-03-04T16:04:00.171374+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.553423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:15.888837+0000","last_change":"2024-03-04T16:04:15.888858+0000","last_active":"2024-03-04T16:04:15.888837+0000","last_peered":"2024-03-04T16:04:15.888837+0000","last_clean":"2024-03-04T16:04:15.888837+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:04:15.888837+0000","last_undegraded":"2024-03-04T16:04:15.888837+0000","last_fullsized":"2024-03-04T16:04:15.888837+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.888765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.888765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.962116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:15.078973+0000","last_change":"2024-03-04T16:04:15.078996+0000","last_active":"2024-03-04T16:04:15.078973+0000","last_peered":"2024-03-04T16:04:15.078973+0000","last_clean":"2024-03-04T16:04:15.078973+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:04:15.078973+0000","last_undegraded":"2024-03-04T16:04:15.078973+0000","last_fullsized":"2024-03-04T16:04:15.078973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.078890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.078890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.844807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095150+0000","last_change":"2024-03-04T16:03:47.979184+0000","last_active":"2024-03-04T16:04:00.095150+0000","last_peered":"2024-03-04T16:04:00.095150+0000","last_clean":"2024-03-04T16:04:00.095150+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:04:00.095150+0000","last_undegraded":"2024-03-04T16:04:00.095150+0000","last_fullsized":"2024-03-04T16:04:00.095150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.979058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.979058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.315882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":91,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:17.030111+0000","last_change":"2024-03-04T16:04:17.030150+0000","last_active":"2024-03-04T16:04:17.030111+0000","last_peered":"2024-03-04T16:04:17.030111+0000","last_clean":"2024-03-04T16:04:17.030111+0000","last_became_active":"2024-03-04T16:02:45.748615+0000","last_became_peered":"2024-03-04T16:02:45.748615+0000","last_unstale":"2024-03-04T16:04:17.030111+0000","last_undegraded":"2024-03-04T16:04:17.030111+0000","last_fullsized":"2024-03-04T16:04:17.030111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:04:17.030053+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:45.396813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:03.048679+0000","last_change":"2024-03-04T16:04:03.048703+0000","last_active":"2024-03-04T16:04:03.048679+0000","last_peered":"2024-03-04T16:04:03.048679+0000","last_clean":"2024-03-04T16:04:03.048679+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:04:03.048679+0000","last_undegraded":"2024-03-04T16:04:03.048679+0000","last_fullsized":"2024-03-04T16:04:03.048679+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.048620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.048620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.405283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.164260+0000","last_change":"2024-03-04T16:04:16.164295+0000","last_active":"2024-03-04T16:04:16.164260+0000","last_peered":"2024-03-04T16:04:16.164260+0000","last_clean":"2024-03-04T16:04:16.164260+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:04:16.164260+0000","last_undegraded":"2024-03-04T16:04:16.164260+0000","last_fullsized":"2024-03-04T16:04:16.164260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.164189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.164189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:27.368351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":315,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170921+0000","last_change":"2024-03-04T16:03:12.851794+0000","last_active":"2024-03-04T16:04:00.170921+0000","last_peered":"2024-03-04T16:04:00.170921+0000","last_clean":"2024-03-04T16:04:00.170921+0000","last_became_active":"2024-03-04T16:02:34.079164+0000","last_became_peered":"2024-03-04T16:02:34.079164+0000","last_unstale":"2024-03-04T16:04:00.170921+0000","last_undegraded":"2024-03-04T16:04:00.170921+0000","last_fullsized":"2024-03-04T16:04:00.170921+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'119","last_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:03:12.851677+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:36.356853+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096256+0000","last_change":"2024-03-04T16:03:57.445763+0000","last_active":"2024-03-04T16:04:00.096256+0000","last_peered":"2024-03-04T16:04:00.096256+0000","last_clean":"2024-03-04T16:04:00.096256+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:04:00.096256+0000","last_undegraded":"2024-03-04T16:04:00.096256+0000","last_fullsized":"2024-03-04T16:04:00.096256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.445659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.445659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.402395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775669+0000","last_change":"2024-03-04T16:03:58.764928+0000","last_active":"2024-03-04T16:04:08.775669+0000","last_peered":"2024-03-04T16:04:08.775669+0000","last_clean":"2024-03-04T16:04:08.775669+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:04:08.775669+0000","last_undegraded":"2024-03-04T16:04:08.775669+0000","last_fullsized":"2024-03-04T16:04:08.775669+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:03:58.764821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.764821+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.352281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172055+0000","last_change":"2024-03-04T16:03:52.917188+0000","last_active":"2024-03-04T16:04:00.172055+0000","last_peered":"2024-03-04T16:04:00.172055+0000","last_clean":"2024-03-04T16:04:00.172055+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:04:00.172055+0000","last_undegraded":"2024-03-04T16:04:00.172055+0000","last_fullsized":"2024-03-04T16:04:00.172055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.917090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.917090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:00.018319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171270+0000","last_change":"2024-03-04T16:03:52.944487+0000","last_active":"2024-03-04T16:04:00.171270+0000","last_peered":"2024-03-04T16:04:00.171270+0000","last_clean":"2024-03-04T16:04:00.171270+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:04:00.171270+0000","last_undegraded":"2024-03-04T16:04:00.171270+0000","last_fullsized":"2024-03-04T16:04:00.171270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.944410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.132708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.738565+0000","last_change":"2024-03-04T16:04:09.738590+0000","last_active":"2024-03-04T16:04:09.738565+0000","last_peered":"2024-03-04T16:04:09.738565+0000","last_clean":"2024-03-04T16:04:09.738565+0000","last_became_active":"2024-03-04T16:02:43.713276+0000","last_became_peered":"2024-03-04T16:02:43.713276+0000","last_unstale":"2024-03-04T16:04:09.738565+0000","last_undegraded":"2024-03-04T16:04:09.738565+0000","last_fullsized":"2024-03-04T16:04:09.738565+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.738490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.738490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.085127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":163,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.147707+0000","last_change":"2024-03-04T16:04:07.872620+0000","last_active":"2024-03-04T16:04:16.147707+0000","last_peered":"2024-03-04T16:04:16.147707+0000","last_clean":"2024-03-04T16:04:16.147707+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:04:16.147707+0000","last_undegraded":"2024-03-04T16:04:16.147707+0000","last_fullsized":"2024-03-04T16:04:16.147707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-04T16:04:07.872489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.872489+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.219006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171663+0000","last_change":"2024-03-04T16:02:45.755785+0000","last_active":"2024-03-04T16:04:00.171663+0000","last_peered":"2024-03-04T16:04:00.171663+0000","last_clean":"2024-03-04T16:04:00.171663+0000","last_became_active":"2024-03-04T16:02:45.755435+0000","last_became_peered":"2024-03-04T16:02:45.755435+0000","last_unstale":"2024-03-04T16:04:00.171663+0000","last_undegraded":"2024-03-04T16:04:00.171663+0000","last_fullsized":"2024-03-04T16:04:00.171663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.974009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172637+0000","last_change":"2024-03-04T16:03:57.296517+0000","last_active":"2024-03-04T16:04:00.172637+0000","last_peered":"2024-03-04T16:04:00.172637+0000","last_clean":"2024-03-04T16:04:00.172637+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:04:00.172637+0000","last_undegraded":"2024-03-04T16:04:00.172637+0000","last_fullsized":"2024-03-04T16:04:00.172637+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.296421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.296421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.272283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170706+0000","last_change":"2024-03-04T16:02:49.785263+0000","last_active":"2024-03-04T16:04:00.170706+0000","last_peered":"2024-03-04T16:04:00.170706+0000","last_clean":"2024-03-04T16:04:00.170706+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:04:00.170706+0000","last_undegraded":"2024-03-04T16:04:00.170706+0000","last_fullsized":"2024-03-04T16:04:00.170706+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.609318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171392+0000","last_change":"2024-03-04T16:02:43.719367+0000","last_active":"2024-03-04T16:04:00.171392+0000","last_peered":"2024-03-04T16:04:00.171392+0000","last_clean":"2024-03-04T16:04:00.171392+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:04:00.171392+0000","last_undegraded":"2024-03-04T16:04:00.171392+0000","last_fullsized":"2024-03-04T16:04:00.171392+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:10.095060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:10.746292+0000","last_change":"2024-03-04T16:04:10.746331+0000","last_active":"2024-03-04T16:04:10.746292+0000","last_peered":"2024-03-04T16:04:10.746292+0000","last_clean":"2024-03-04T16:04:10.746292+0000","last_became_active":"2024-03-04T16:02:45.744172+0000","last_became_peered":"2024-03-04T16:02:45.744172+0000","last_unstale":"2024-03-04T16:04:10.746292+0000","last_undegraded":"2024-03-04T16:04:10.746292+0000","last_fullsized":"2024-03-04T16:04:10.746292+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:04:10.746233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.746233+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:36.591472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.087204+0000","last_change":"2024-03-04T16:03:50.770141+0000","last_active":"2024-03-04T16:04:00.087204+0000","last_peered":"2024-03-04T16:04:00.087204+0000","last_clean":"2024-03-04T16:04:00.087204+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:04:00.087204+0000","last_undegraded":"2024-03-04T16:04:00.087204+0000","last_fullsized":"2024-03-04T16:04:00.087204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.770048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.770048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.417482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170347+0000","last_change":"2024-03-04T16:02:49.797759+0000","last_active":"2024-03-04T16:04:00.170347+0000","last_peered":"2024-03-04T16:04:00.170347+0000","last_clean":"2024-03-04T16:04:00.170347+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:04:00.170347+0000","last_undegraded":"2024-03-04T16:04:00.170347+0000","last_fullsized":"2024-03-04T16:04:00.170347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:13.105311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:15.206361+0000","last_change":"2024-03-04T16:04:15.206400+0000","last_active":"2024-03-04T16:04:15.206361+0000","last_peered":"2024-03-04T16:04:15.206361+0000","last_clean":"2024-03-04T16:04:15.206361+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:04:15.206361+0000","last_undegraded":"2024-03-04T16:04:15.206361+0000","last_fullsized":"2024-03-04T16:04:15.206361+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.206292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.206292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.696036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:14.060777+0000","last_change":"2024-03-04T16:04:14.060813+0000","last_active":"2024-03-04T16:04:14.060777+0000","last_peered":"2024-03-04T16:04:14.060777+0000","last_clean":"2024-03-04T16:04:14.060777+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:04:14.060777+0000","last_undegraded":"2024-03-04T16:04:14.060777+0000","last_fullsized":"2024-03-04T16:04:14.060777+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:04:14.060715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.060715+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:30.595410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775516+0000","last_change":"2024-03-04T16:02:47.777918+0000","last_active":"2024-03-04T16:04:08.775516+0000","last_peered":"2024-03-04T16:04:08.775516+0000","last_clean":"2024-03-04T16:04:08.775516+0000","last_became_active":"2024-03-04T16:02:47.777462+0000","last_became_peered":"2024-03-04T16:02:47.777462+0000","last_unstale":"2024-03-04T16:04:08.775516+0000","last_undegraded":"2024-03-04T16:04:08.775516+0000","last_fullsized":"2024-03-04T16:04:08.775516+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171829+0000","last_change":"2024-03-04T16:03:56.858487+0000","last_active":"2024-03-04T16:04:00.171829+0000","last_peered":"2024-03-04T16:04:00.171829+0000","last_clean":"2024-03-04T16:04:00.171829+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:04:00.171829+0000","last_undegraded":"2024-03-04T16:04:00.171829+0000","last_fullsized":"2024-03-04T16:04:00.171829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.858376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.858376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.769864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.701244+0000","last_change":"2024-03-04T16:04:16.701272+0000","last_active":"2024-03-04T16:04:16.701244+0000","last_peered":"2024-03-04T16:04:16.701244+0000","last_clean":"2024-03-04T16:04:16.701244+0000","last_became_active":"2024-03-04T16:02:43.712287+0000","last_became_peered":"2024-03-04T16:02:43.712287+0000","last_unstale":"2024-03-04T16:04:16.701244+0000","last_undegraded":"2024-03-04T16:04:16.701244+0000","last_fullsized":"2024-03-04T16:04:16.701244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:16.701175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.701175+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.474917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:06.515895+0000","last_change":"2024-03-04T16:04:06.515940+0000","last_active":"2024-03-04T16:04:06.515895+0000","last_peered":"2024-03-04T16:04:06.515895+0000","last_clean":"2024-03-04T16:04:06.515895+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:04:06.515895+0000","last_undegraded":"2024-03-04T16:04:06.515895+0000","last_fullsized":"2024-03-04T16:04:06.515895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.515823+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.602986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.344572+0000","last_change":"2024-03-04T16:04:09.344572+0000","last_active":"2024-03-04T16:04:09.344572+0000","last_peered":"2024-03-04T16:04:09.344572+0000","last_clean":"2024-03-04T16:04:09.344572+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:04:09.344572+0000","last_undegraded":"2024-03-04T16:04:09.344572+0000","last_fullsized":"2024-03-04T16:04:09.344572+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.344448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.344448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.358656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171503+0000","last_change":"2024-03-04T16:02:49.789412+0000","last_active":"2024-03-04T16:04:00.171503+0000","last_peered":"2024-03-04T16:04:00.171503+0000","last_clean":"2024-03-04T16:04:00.171503+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:04:00.171503+0000","last_undegraded":"2024-03-04T16:04:00.171503+0000","last_fullsized":"2024-03-04T16:04:00.171503+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:16.799791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096671+0000","last_change":"2024-03-04T16:02:43.716171+0000","last_active":"2024-03-04T16:04:00.096671+0000","last_peered":"2024-03-04T16:04:00.096671+0000","last_clean":"2024-03-04T16:04:00.096671+0000","last_became_active":"2024-03-04T16:02:43.715346+0000","last_became_peered":"2024-03-04T16:02:43.715346+0000","last_unstale":"2024-03-04T16:04:00.096671+0000","last_undegraded":"2024-03-04T16:04:00.096671+0000","last_fullsized":"2024-03-04T16:04:00.096671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095990+0000","last_change":"2024-03-04T16:02:45.749977+0000","last_active":"2024-03-04T16:04:00.095990+0000","last_peered":"2024-03-04T16:04:00.095990+0000","last_clean":"2024-03-04T16:04:00.095990+0000","last_became_active":"2024-03-04T16:02:45.749633+0000","last_became_peered":"2024-03-04T16:02:45.749633+0000","last_unstale":"2024-03-04T16:04:00.095990+0000","last_undegraded":"2024-03-04T16:04:00.095990+0000","last_fullsized":"2024-03-04T16:04:00.095990+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:03.646959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170756+0000","last_change":"2024-03-04T16:02:47.768154+0000","last_active":"2024-03-04T16:04:00.170756+0000","last_peered":"2024-03-04T16:04:00.170756+0000","last_clean":"2024-03-04T16:04:00.170756+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:04:00.170756+0000","last_undegraded":"2024-03-04T16:04:00.170756+0000","last_fullsized":"2024-03-04T16:04:00.170756+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.581920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172282+0000","last_change":"2024-03-04T16:02:49.792150+0000","last_active":"2024-03-04T16:04:00.172282+0000","last_peered":"2024-03-04T16:04:00.172282+0000","last_clean":"2024-03-04T16:04:00.172282+0000","last_became_active":"2024-03-04T16:02:49.791851+0000","last_became_peered":"2024-03-04T16:02:49.791851+0000","last_unstale":"2024-03-04T16:04:00.172282+0000","last_undegraded":"2024-03-04T16:04:00.172282+0000","last_fullsized":"2024-03-04T16:04:00.172282+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:15.262236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775678+0000","last_change":"2024-03-04T16:02:47.768859+0000","last_active":"2024-03-04T16:04:08.775678+0000","last_peered":"2024-03-04T16:04:08.775678+0000","last_clean":"2024-03-04T16:04:08.775678+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:04:08.775678+0000","last_undegraded":"2024-03-04T16:04:08.775678+0000","last_fullsized":"2024-03-04T16:04:08.775678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.095980+0000","last_change":"2024-03-04T16:03:58.462106+0000","last_active":"2024-03-04T16:04:00.095980+0000","last_peered":"2024-03-04T16:04:00.095980+0000","last_clean":"2024-03-04T16:04:00.095980+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:04:00.095980+0000","last_undegraded":"2024-03-04T16:04:00.095980+0000","last_fullsized":"2024-03-04T16:04:00.095980+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.462018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:08.477439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171988+0000","last_change":"2024-03-04T16:03:46.963168+0000","last_active":"2024-03-04T16:04:00.171988+0000","last_peered":"2024-03-04T16:04:00.171988+0000","last_clean":"2024-03-04T16:04:00.171988+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:04:00.171988+0000","last_undegraded":"2024-03-04T16:04:00.171988+0000","last_fullsized":"2024-03-04T16:04:00.171988+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:46.963065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.963065+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.801631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.096277+0000","last_change":"2024-03-04T16:02:49.794003+0000","last_active":"2024-03-04T16:04:00.096277+0000","last_peered":"2024-03-04T16:04:00.096277+0000","last_clean":"2024-03-04T16:04:00.096277+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:04:00.096277+0000","last_undegraded":"2024-03-04T16:04:00.096277+0000","last_fullsized":"2024-03-04T16:04:00.096277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.636557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:13.194780+0000","last_change":"2024-03-04T16:04:13.194800+0000","last_active":"2024-03-04T16:04:13.194780+0000","last_peered":"2024-03-04T16:04:13.194780+0000","last_clean":"2024-03-04T16:04:13.194780+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:04:13.194780+0000","last_undegraded":"2024-03-04T16:04:13.194780+0000","last_fullsized":"2024-03-04T16:04:13.194780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:13.194723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.194723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.282552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.170870+0000","last_change":"2024-03-04T16:02:43.705402+0000","last_active":"2024-03-04T16:04:00.170870+0000","last_peered":"2024-03-04T16:04:00.170870+0000","last_clean":"2024-03-04T16:04:00.170870+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:04:00.170870+0000","last_undegraded":"2024-03-04T16:04:00.170870+0000","last_fullsized":"2024-03-04T16:04:00.170870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:12.029120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":70,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.172915+0000","last_change":"2024-03-04T16:02:45.746393+0000","last_active":"2024-03-04T16:04:00.172915+0000","last_peered":"2024-03-04T16:04:00.172915+0000","last_clean":"2024-03-04T16:04:00.172915+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:04:00.172915+0000","last_undegraded":"2024-03-04T16:04:00.172915+0000","last_fullsized":"2024-03-04T16:04:00.172915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:09.554587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.063138+0000","last_change":"2024-03-04T16:04:07.063161+0000","last_active":"2024-03-04T16:04:07.063138+0000","last_peered":"2024-03-04T16:04:07.063138+0000","last_clean":"2024-03-04T16:04:07.063138+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:04:07.063138+0000","last_undegraded":"2024-03-04T16:04:07.063138+0000","last_fullsized":"2024-03-04T16:04:07.063138+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.063063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.063063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.152553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.976020+0000","last_change":"2024-03-04T16:04:04.976057+0000","last_active":"2024-03-04T16:04:04.976020+0000","last_peered":"2024-03-04T16:04:04.976020+0000","last_clean":"2024-03-04T16:04:04.976020+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:04:04.976020+0000","last_undegraded":"2024-03-04T16:04:04.976020+0000","last_fullsized":"2024-03-04T16:04:04.976020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.975944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.975944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.116230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:07.763022+0000","last_change":"2024-03-04T16:04:07.763042+0000","last_active":"2024-03-04T16:04:07.763022+0000","last_peered":"2024-03-04T16:04:07.763022+0000","last_clean":"2024-03-04T16:04:07.763022+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:04:07.763022+0000","last_undegraded":"2024-03-04T16:04:07.763022+0000","last_fullsized":"2024-03-04T16:04:07.763022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.762953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:37.004167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":87,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:09.533334+0000","last_change":"2024-03-04T16:04:09.533365+0000","last_active":"2024-03-04T16:04:09.533334+0000","last_peered":"2024-03-04T16:04:09.533334+0000","last_clean":"2024-03-04T16:04:09.533334+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:04:09.533334+0000","last_undegraded":"2024-03-04T16:04:09.533334+0000","last_fullsized":"2024-03-04T16:04:09.533334+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:04:09.533267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.533267+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.228405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:04.509705+0000","last_change":"2024-03-04T16:04:04.509748+0000","last_active":"2024-03-04T16:04:04.509705+0000","last_peered":"2024-03-04T16:04:04.509705+0000","last_clean":"2024-03-04T16:04:04.509705+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:04:04.509705+0000","last_undegraded":"2024-03-04T16:04:04.509705+0000","last_fullsized":"2024-03-04T16:04:04.509705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.509608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.509608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.020538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"75'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:08.775499+0000","last_change":"2024-03-04T16:02:47.770597+0000","last_active":"2024-03-04T16:04:08.775499+0000","last_peered":"2024-03-04T16:04:08.775499+0000","last_clean":"2024-03-04T16:04:08.775499+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:04:08.775499+0000","last_undegraded":"2024-03-04T16:04:08.775499+0000","last_fullsized":"2024-03-04T16:04:08.775499+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:16.373359+0000","last_change":"2024-03-04T16:04:16.373386+0000","last_active":"2024-03-04T16:04:16.373359+0000","last_peered":"2024-03-04T16:04:16.373359+0000","last_clean":"2024-03-04T16:04:16.373359+0000","last_became_active":"2024-03-04T16:02:43.714792+0000","last_became_peered":"2024-03-04T16:02:43.714792+0000","last_unstale":"2024-03-04T16:04:16.373359+0000","last_undegraded":"2024-03-04T16:04:16.373359+0000","last_fullsized":"2024-03-04T16:04:16.373359+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.373275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.726905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-04T16:04:00.171318+0000","last_change":"2024-03-04T16:02:45.744111+0000","last_active":"2024-03-04T16:04:00.171318+0000","last_peered":"2024-03-04T16:04:00.171318+0000","last_clean":"2024-03-04T16:04:00.171318+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:04:00.171318+0000","last_undegraded":"2024-03-04T16:04:00.171318+0000","last_fullsized":"2024-03-04T16:04:00.171318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:07.765686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070487,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1413120,"data_stored":946996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364827,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32532,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712620,"statfs":{"total":95995035648,"available":95961722880,"internally_reserved":0,"allocated":1388544,"data_stored":944684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":856064,"data_stored":356088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28396,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716756,"statfs":{"total":95995035648,"available":95965958144,"internally_reserved":0,"allocated":1413120,"data_stored":946340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27788,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717364,"statfs":{"total":95995035648,"available":95966580736,"internally_reserved":0,"allocated":790528,"data_stored":354816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":356082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":355306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626465,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":354642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:04:19.885 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-04T16:04:19.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump 2024-03-04T16:04:20.129 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:20.231 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:19 smithi176 podman[63937]: 2024-03-04 16:04:19.817390264 +0000 UTC m=+0.017737015 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-04T16:04:20.231 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:19 smithi176 podman[63937]: 2024-03-04 16:04:19.92061815 +0000 UTC m=+0.120964881 container create e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5) 2024-03-04T16:04:20.231 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 podman[63937]: 2024-03-04 16:04:20.156336503 +0000 UTC m=+0.356683252 container init e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, vendor=Red Hat, Inc., maintainer=Paul Cuzner , name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, release=239.1651231664, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-04T16:04:20.232 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 podman[63937]: 2024-03-04 16:04:20.159043854 +0000 UTC m=+0.359390598 container start e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, distribution-scope=public, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, maintainer=Paul Cuzner , vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-04T16:04:20.232 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 bash[63937]: e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 2024-03-04T16:04:20.482 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 systemd[1]: Started Ceph grafana.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.308887713Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309086102Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309104667Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309111696Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309117852Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309123787Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.3091296Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.30913534Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309141147Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309146858Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309153091Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-04T16:04:20.483 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309158511Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309164121Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=settings t=2024-03-04T16:04:20.309169736Z level=info msg="App mode production" 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore t=2024-03-04T16:04:20.309212256Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore t=2024-03-04T16:04:20.309227735Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.310078883Z level=info msg="Starting DB migrations" 2024-03-04T16:04:20.484 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.31015145Z level=info msg="Executing migration" id="create migration_log table" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.310726905Z level=info msg="Executing migration" id="create user table" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.31115213Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.313860312Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.31427991Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.317194049Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.319615512Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.320339822Z level=info msg="Executing migration" id="create user table v2" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.323193795Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-04T16:04:20.485 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.326014029Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.328722642Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.329084434Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.331755242Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.334565234Z level=info msg="Executing migration" id="Update user table charset" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.337233626Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.337724459Z level=info msg="Executing migration" id="Add missing user data" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.338044493Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.338485254Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.338888495Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.339266912Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.340395682Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.340847068Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-04T16:04:20.486 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.341320057Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.341746698Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.342166003Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.342591635Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.342894148Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.343257661Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.343660665Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.344024184Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.344376593Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.345030476Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.345433363Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.345855387Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.346223844Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.346633778Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-04T16:04:20.487 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.347078186Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.347424997Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.347804021Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.348115187Z level=info msg="Executing migration" id="create star table" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.348512087Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.348951001Z level=info msg="Executing migration" id="create org table v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.349339512Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.349770588Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.350143044Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.35056772Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.351030269Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.351451323Z level=info msg="Executing migration" id="Update org table charset" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.351786137Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-04T16:04:20.488 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.352074221Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.352380352Z level=info msg="Executing migration" id="create dashboard table" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.352802251Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.353226217Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.353934916Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.354385248Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.354908343Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.355695243Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.356604082Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.357062519Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.357896209Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.358394136Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.359186813Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.359616508Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-04T16:04:20.489 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.359960596Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.360535399Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.361382861Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.362223845Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.363210293Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.363874687Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.364765471Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.365361485Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.365699731Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.36598235Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.366569881Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.367139907Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.368099034Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.368895075Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-04T16:04:20.490 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.369676712Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.370166226Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.373066393Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.37571694Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.378647481Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.379101836Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.38206212Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.384960651Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.387748403Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.390508463Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.393445717Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.39613935Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.398956563Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.399522114Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-04T16:04:20.491 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.402367802Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.405034417Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.407976388Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.408573656Z level=info msg="Executing migration" id="create data_source table" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.411502287Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.414077957Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-04T16:04:20.492 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.414529482Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.414931998Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.417755104Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.418976498Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.420499268Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.423209664Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.423730891Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.426497571Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.427033371Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.429216729Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.431918924Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-04T16:04:20.493 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.432261502Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.434975171Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.437616956Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.440229906Z level=info msg="Executing migration" id="Add uid column" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.440936375Z level=info msg="Executing migration" id="Update uid value" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.44364175Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.444122371Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.446604998Z level=info msg="Executing migration" id="create api_key table" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.447318052Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.447781348Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.448776321Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.449431707Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.450261767Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-04T16:04:20.494 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.451086446Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.451900776Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.4529698Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.453800814Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.454741527Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.455408521Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.456222902Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.456565457Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.456978686Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.457251497Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.457810467Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.458357695Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.45870964Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.459265758Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-04T16:04:20.495 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.459883051Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.46024887Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.46067478Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.461137619Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.46157288Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.462047642Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.463068371Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.463751553Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.464655685Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.465433108Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.466322994Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.467151702Z level=info msg="Executing migration" id="create quota table v1" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.468009255Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-04T16:04:20.496 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.468967178Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.469647525Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.470058012Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.470928229Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.473822478Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.476648243Z level=info msg="Executing migration" id="create session table" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.477128922Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.479777508Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.482247867Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.48498183Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-04T16:04:20.497 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.487680281Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: from='client.14793 v1:172.21.15.38:0/618214226' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.1b deep-scrub starts 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: 5.1b deep-scrub ok 2024-03-04T16:04:20.733 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.734 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.734 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.490165715Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.492801779Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.495394258Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.49816089Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.500863668Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.503413257Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.505782821Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.508492016Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.51123559Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.513680339Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.516394246Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.517346794Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.518063648Z level=info msg="Executing migration" id="create alert table v1" 2024-03-04T16:04:20.735 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.51900406Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.519744731Z level=info msg="Executing migration" id="add index alert state" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.520741337Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.521458465Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.52230314Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.52310765Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.523995821Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.525383246Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.526190149Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.526989321Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.527944806Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.528688437Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.529648382Z level=info msg="Executing migration" id="Add column is_default" 2024-03-04T16:04:20.736 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.530360605Z level=info msg="Executing migration" id="Add column frequency" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.53118159Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.532174422Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.532982761Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.533966912Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.534674171Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.535635422Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.536525311Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.537276958Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.538228698Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.538957057Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.539890528Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.540727988Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-04T16:04:20.737 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.541366318Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.542212561Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.543163555Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.543855638Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.544817952Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.545485546Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.546470889Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.549272776Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.552283946Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.554792152Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.557629562Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.560607793Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.563364012Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-04T16:04:20.738 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.566140296Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-04T16:04:20.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.568882131Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-04T16:04:20.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.57150646Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-04T16:04:20.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.574319062Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-04T16:04:20.739 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.5773561Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.579987556Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.582795087Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.585515883Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.588544772Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.591346942Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.593936126Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.596658017Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.599380204Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.602136887Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-04T16:04:20.740 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.604605645Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.607511968Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.610085266Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.612730513Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.615420282Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.618226826Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.620634124Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.62343092Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.626249243Z level=info msg="Executing migration" id="Move region to single row" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.628680185Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.631498058Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.634341111Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.637100649Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.639641119Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-04T16:04:20.741 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.642207535Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.643091332Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.644082804Z level=info msg="Executing migration" id="create test_data table" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.644748299Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.645506657Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.646504321Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.647272174Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.648043927Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.64880458Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.649116902Z level=info msg="Executing migration" id="create team table" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.649678423Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.650506851Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.651433576Z level=info msg="Executing migration" id="create team member table" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.652124308Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-04T16:04:20.742 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.653094769Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.653835123Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.654701093Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.655596866Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.656461371Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.657359337Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.65820967Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.659027728Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.659902378Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.660898327Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.661604732Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.662599757Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.663337191Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-04T16:04:20.743 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.664248408Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.665133711Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.665660171Z level=info msg="Executing migration" id="create tag table" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.666425765Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.667403366Z level=info msg="Executing migration" id="create login attempt table" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.668141377Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.668591977Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.669539962Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.671466212Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.674507712Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.675058665Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.677742986Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.678134738Z level=info msg="Executing migration" id="create user auth table" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.680675287Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-04T16:04:20.744 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.683480605Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.683791112Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.686468661Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.689341182Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.690114987Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.692986093Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.695733708Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.696548176Z level=info msg="Executing migration" id="create server_lock table" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.699339724Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.702138453Z level=info msg="Executing migration" id="create user auth token table" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.704926225Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.707463496Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.710370894Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.710825273Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-04T16:04:20.745 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.71350507Z level=info msg="Executing migration" id="create cache_data table" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.716196642Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.718891567Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.719535372Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.722383691Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.724961892Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.725332213Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.72581284Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.728394579Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.72887719Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.729150124Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.729596393Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.730027448Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.730470863Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-04T16:04:20.746 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.730902111Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.731754362Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.732193379Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.73251533Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.732921979Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.733368975Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.73390617Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.734201994Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.734632341Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.735053872Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.735478469Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.735969271Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.736834975Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.737276574Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-04T16:04:20.747 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.73775787Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-04T16:04:20.895 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: from='client.14793 v1:172.21.15.38:0/618214226' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.1b deep-scrub starts 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: 5.1b deep-scrub ok 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: from='client.14793 v1:172.21.15.38:0/618214226' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.1b deep-scrub starts 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: 5.1b deep-scrub ok 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:20.897 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:20 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.741991175Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.74613823Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.74657491Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.746997782Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.747833556Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.748236706Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.748664123Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.749072696Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.749506677Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.74978963Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.750775467Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-04T16:04:21.018 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.751593955Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.752528001Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.752933369Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.753332041Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.754123324Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.75494866Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-04T16:04:21.019 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.75537881Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.756188347Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.757004556Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.757309184Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.757749425Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.758163542Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.758598273Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.758881712Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.759798343Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.760663033Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-04T16:04:21.020 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.761519931Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.762339664Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.763165014Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.763494342Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.763872483Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.764703642Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.764994953Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.76588693Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.768719335Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.77140585Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.774060243Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.774526657Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.775408886Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-04T16:04:21.021 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.776210278Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.777208744Z level=info msg="Executing migration" id="create alert_image table" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.777849759Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.778705315Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.779453005Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.779878305Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.780282686Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.780372676Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.780651705Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.781118534Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.7815729Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.781696791Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.78181748Z level=info msg="alerts found to migrate" alerts=0 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.782216381Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-04T16:04:21.022 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.783079685Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.783591053Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.784401894Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.78519607Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.78613889Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.786638212Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.787468514Z level=info msg="Executing migration" id="create data_keys table" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.787905635Z level=info msg="Executing migration" id="create secrets table" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.788364502Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.792597372Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.793500476Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.79448292Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.798902232Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-04T16:04:21.023 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.803183355Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.803642656Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.804455899Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.80540497Z level=info msg="Executing migration" id="create permission table" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.806150729Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.806647787Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.80743372Z level=info msg="Executing migration" id="create role table" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.80838313Z level=info msg="Executing migration" id="add column display_name" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.809307792Z level=info msg="Executing migration" id="add column group_name" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.810238258Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.81313255Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.815820007Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.818674045Z level=info msg="Executing migration" id="create team role table" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.819093271Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-04T16:04:21.024 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.821924066Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.824718484Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.827523421Z level=info msg="Executing migration" id="create user role table" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.827929982Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.830712998Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.833451446Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.836240026Z level=info msg="Executing migration" id="create builtin role table" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.83672522Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.839499106Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.842229554Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.843230879Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-04T16:04:21.025 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.845248052Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.845757571Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.846565726Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.84704741Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.849616493Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.850123674Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.85292112Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-04T16:04:21.026 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.855909268Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.858937789Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.859416712Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.862099009Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.864942868Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.867679729Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.868010791Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.870709519Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.87119572Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.873543101Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.876267567Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.876611858Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.879406446Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.882098314Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.884667168Z level=info msg="Executing migration" id="add correlation config column" 2024-03-04T16:04:21.027 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.885834403Z level=info msg="Executing migration" id="create entity_events table" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.888493302Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.891188511Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.891284729Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.893885535Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.89399029Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.894295615Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.896808315Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.899594873Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.902360988Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.902892195Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.905565668Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T16:04:21.028 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.906172143Z level=info msg="Executing migration" id="Drop public config table" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.908194902Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.910949974Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.911422812Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.914054399Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.916848429Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.920055489Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.921072059Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.922079864Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.922924666Z level=info msg="Executing migration" id="add share column" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.92398498Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.924473755Z level=info msg="Executing migration" id="create file table" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.925314167Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.926221464Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-04T16:04:21.029 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.926869668Z level=info msg="Executing migration" id="create file_meta table" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.927279144Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.928181733Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.928851348Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.929646822Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.929999849Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.930456429Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.931498328Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.932271546Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.933047945Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.93351413Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.934454669Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.935341509Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-04T16:04:21.030 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.936538732Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.937563834Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.938399134Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.948178885Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.949017988Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.949814151Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.953303216Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.95408919Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.954943314Z level=info msg="Executing migration" id="create folder table" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.955400089Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.956380042Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.957112247Z level=info msg="Executing migration" id="Update folder title length" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.957958693Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=migrator t=2024-03-04T16:04:20.958549865Z level=info msg="migrations completed" performed=485 skipped=0 duration=648.416647ms 2024-03-04T16:04:21.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore t=2024-03-04T16:04:20.959156339Z level=info msg="Created default organization" 2024-03-04T16:04:21.032 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=plugin.loader t=2024-03-04T16:04:21.017237678Z level=info msg="Plugin registered" pluginID=input 2024-03-04T16:04:21.294 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=plugin.loader t=2024-03-04T16:04:21.044875073Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-04T16:04:21.294 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=plugin.loader t=2024-03-04T16:04:21.044895137Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-04T16:04:21.294 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=secrets t=2024-03-04T16:04:21.04492676Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-04T16:04:21.295 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=query_data t=2024-03-04T16:04:21.046944502Z level=info msg="Query Service initialization" 2024-03-04T16:04:21.295 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=live.push_http t=2024-03-04T16:04:21.049306143Z level=info msg="Live Push Gateway initialization" 2024-03-04T16:04:21.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: Deploying daemon node-exporter.a on smithi038 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: Deploying daemon node-exporter.a on smithi038 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.776 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.777 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.777 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.777 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.777 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:21.777 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:21 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: Deploying daemon node-exporter.a on smithi038 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=infra.usagestats.collector t=2024-03-04T16:04:21.705789511Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=provisioning.datasources t=2024-03-04T16:04:21.706974104Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=provisioning.datasources t=2024-03-04T16:04:21.722080753Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=provisioning.alerting t=2024-03-04T16:04:21.734500344Z level=info msg="starting to provision alerting" 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=provisioning.alerting t=2024-03-04T16:04:21.734527373Z level=info msg="finished to provision alerting" 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=grafanaStorageLogger t=2024-03-04T16:04:21.734965225Z level=info msg="storage starting" 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=ngalert.state.manager t=2024-03-04T16:04:21.734987774Z level=info msg="Warming state cache for startup" 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=http.server t=2024-03-04T16:04:21.737608028Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.746582923Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T16:04:21.860 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=ngalert.state.manager t=2024-03-04T16:04:21.747111649Z level=info msg="State cache has been initialized" states=0 duration=12.121634ms 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=ticker t=2024-03-04T16:04:21.747191585Z level=info msg=starting first_tick=2024-03-04T16:04:30Z 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=ngalert.multiorg.alertmanager t=2024-03-04T16:04:21.747252137Z level=info msg="starting MultiOrg Alertmanager" 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.757624432Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.762236599Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.768375397Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.781059547Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-04T16:04:21.861 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:04:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:04:21.791315317Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-04T16:04:22.007 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 92 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:fsid c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-03-04T15:57:15.255572+0000 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-03-04T16:04:21.691331+0000 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 18 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client luminous 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-03-04T16:04:22.008 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 8 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 8 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6801/1080993770 v1:172.21.15.38:6802/1080993770 exists,up 99ecf38d-7bc9-412d-a194-f3d9eed8e244 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6805/2331598100 v1:172.21.15.38:6806/2331598100 exists,up f8c66eb4-4966-45ce-af41-b6d01713eb51 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6809/1898914368 v1:172.21.15.38:6810/1898914368 exists,up afb2dd7e-d27a-4536-bb80-e5724b39ffd3 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 26 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6813/552035408 v1:172.21.15.38:6814/552035408 exists,up 13f08b82-22d5-4466-9ad4-27ffa15ed7c4 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 32 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6800/3997489718 v1:172.21.15.176:6801/3997489718 exists,up a58d697b-e1de-44cc-b791-8a8d68fd01f3 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 42 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6804/1267814386 v1:172.21.15.176:6805/1267814386 exists,up 7adb17cc-4843-401e-a7fd-b726c4a99a06 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 50 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6808/587062426 v1:172.21.15.176:6809/587062426 exists,up 0e12d14b-c02d-4acf-9784-8680f8ee40f4 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 59 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6812/1019151537 v1:172.21.15.176:6813/1019151537 exists,up 1f348507-1649-4e25-b5f5-cb67c6cd045e 2024-03-04T16:04:22.009 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.14 [1,2] 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.1b [1,2] 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.1f [1,2] 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3142726897 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/452578506 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/501976220 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1665273468 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/500449596 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/4073253154 expires 2024-03-05T16:03:21.264346+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/175394725 expires 2024-03-05T15:58:15.846725+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/1249306019 expires 2024-03-05T15:58:15.846725+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/2156780629 expires 2024-03-05T15:57:48.640902+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1997350354 expires 2024-03-05T15:57:48.640902+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2476964715 expires 2024-03-05T15:58:15.846725+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1178105870 expires 2024-03-05T15:57:48.640902+0000 2024-03-04T16:04:22.010 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3751720132 expires 2024-03-05T15:57:48.640902+0000 2024-03-04T16:04:22.011 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2774521881 expires 2024-03-05T15:58:15.846725+0000 2024-03-04T16:04:22.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:22.713 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:22 smithi038 bash[74499]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T16:04:22.920 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6801/1080993770 v1:172.21.15.38:6802/1080993770 exists,up 99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'osd.1 up in weight 1 up_from 13 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6805/2331598100 v1:172.21.15.38:6806/2331598100 exists,up f8c66eb4-4966-45ce-af41-b6d01713eb51', 'osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6809/1898914368 v1:172.21.15.38:6810/1898914368 exists,up afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'osd.3 up in weight 1 up_from 26 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.38:6813/552035408 v1:172.21.15.38:6814/552035408 exists,up 13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'osd.4 up in weight 1 up_from 32 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6800/3997489718 v1:172.21.15.176:6801/3997489718 exists,up a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'osd.5 up in weight 1 up_from 42 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6804/1267814386 v1:172.21.15.176:6805/1267814386 exists,up 7adb17cc-4843-401e-a7fd-b726c4a99a06', 'osd.6 up in weight 1 up_from 50 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6808/587062426 v1:172.21.15.176:6809/587062426 exists,up 0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'osd.7 up in weight 1 up_from 59 up_thru 73 down_at 0 last_clean_interval [0,0) v1:172.21.15.176:6812/1019151537 v1:172.21.15.176:6813/1019151537 exists,up 1f348507-1649-4e25-b5f5-cb67c6cd045e'] 2024-03-04T16:04:22.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon mon.a config show 2024-03-04T16:04:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:04:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:04:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-04T16:04:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/24669237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-04T16:04:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-04T16:04:22.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:04:22.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:04:22.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:04:22.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/24669237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T16:04:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:04:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: osdmap e92: 8 total, 8 up, 8 in 2024-03-04T16:04:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/24669237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-04T16:04:23.194 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:23.732 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:04:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:04:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:04:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[29779]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:04:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:23 smithi038 ceph-mon[38113]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:04:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:24.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:04:24.122 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:04:24.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:04:24.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:04:24.123 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:23 smithi176 ceph-mon[33619]: osdmap e93: 8 total, 8 up, 8 in 2024-03-04T16:04:24.325 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:23 smithi038 bash[74499]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T16:04:24.325 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:24 smithi038 bash[74499]: Writing manifest to image destination 2024-03-04T16:04:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[38113]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:04:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:04:24.994 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:24 smithi038 podman[74499]: 2024-03-04 16:04:24.824472268 +0000 UTC m=+2.949417547 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T16:04:24.994 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:24 smithi038 podman[74499]: 2024-03-04 16:04:24.931417382 +0000 UTC m=+3.056362695 container create aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[29779]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:04:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:04:24.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:24 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:04:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:24 smithi176 ceph-mon[33619]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:24 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:04:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:24 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:04:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:24 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:04:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:24 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:04:25.353 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 podman[74499]: 2024-03-04 16:04:25.150434431 +0000 UTC m=+3.275379703 container init aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:25.353 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 podman[74499]: 2024-03-04 16:04:25.152999843 +0000 UTC m=+3.277945107 container start aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.156Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.156Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.157Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.157Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.157Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.157Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T16:04:25.354 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T16:04:25.355 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T16:04:25.356 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T16:04:25.357 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a[74657]: ts=2024-03-04T16:04:25.158Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 bash[74499]: aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 2024-03-04T16:04:25.358 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:04:25 smithi038 systemd[1]: Started Ceph node-exporter.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:04:26.303 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mon.a", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster": "ceph", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "admin_socket_mode": "", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_client_required": "cephx, none", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_cluster_required": "cephx", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_debug": "true", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-04T16:04:26.304 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_required": "cephx", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "auth_supported": "", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio": "true", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_poll_ms": "250", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_aio_reap_max": "16", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_async_discard": "false", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_block_size": "4096", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio": "false", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-04T16:04:26.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_debug_inflight_ios": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_enable_discard": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry": "3", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash": "0", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_hipri": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bdev_type": "", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_alloc_size": "1048576", 2024-03-04T16:04:26.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_allocator": "hybrid", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_buffered_io": "true", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_for_zeros": "false", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_compact_log_sync": "false", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_log_runway": "4194304", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_max_prefetch": "1048576", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_flush_size": "524288", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_min_log_runway": "1048576", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery": "false", 2024-03-04T16:04:26.307 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_sync_write": "false", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocation_from_file": "true", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_allocator": "hybrid", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-04T16:04:26.308 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_create": "true", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_create": "false", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_path": "", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_db_size": "0", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_path": "", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_preallocate_file": "false", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_size": "107374182400", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_create": "false", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_path": "", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_block_wal_size": "100663296", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs": "true", 2024-03-04T16:04:26.309 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune": "true", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size": "0", 2024-03-04T16:04:26.310 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_type": "2q", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_clone_cow": "true", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-04T16:04:26.311 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_mode": "none", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_csum_type": "crc32c", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_freelist": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-04T16:04:26.312 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_small_allocations": "0", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_read": "true", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_default_buffered_write": "false", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-04T16:04:26.313 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fail_eio": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount": "false", 2024-03-04T16:04:26.314 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount": "false", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_ignore_data_csum": "false", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-04T16:04:26.315 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_log_op_age": "5.000000", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_alloc_size": "0", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size": "0", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_max_deferred_txc": "32", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size": "0", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_nid_prealloc": "1024", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-04T16:04:26.316 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_retry_disk_reads": "3", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cf": "true", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_spdk_mem": "512", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-04T16:04:26.317 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_tracing": "false", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-04T16:04:26.318 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_zero_block_detection": "false", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-04T16:04:26.319 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_signatures": "false", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_cluster_require_version": "2", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_signatures": "false", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_require_version": "2", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_signatures": "false", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_service_require_version": "2", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "cephx_sign_messages": "true", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "chdir": "", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_acl_type": "", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_asio_thread_count": "2", 2024-03-04T16:04:26.320 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_mid": "0.750000", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_cache_size": "16384", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_caps_release_delay": "5", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_check_pool_perm": "true", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_force_sync_read": "false", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_getattr_caps": "false", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_debug_inject_tick_delay": "0", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_die_on_failed_remount": "false", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_dirsize_rbytes": "true", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_force_lazyio": "false", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_fs": "", 2024-03-04T16:04:26.321 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_inject_release_failure": "false", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_inline_size": "4096", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mds_namespace": "", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_metadata": "", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_gid": "-1", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_timeout": "300", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mount_uid": "-1", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_mountpoint": "/", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_notify_timeout": "10", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc": "true", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty": "104857600", 2024-03-04T16:04:26.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_max_objects": "1000", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_size": "209715200", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_oc_target_dirty": "8388608", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_permissions": "true", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota": "true", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_quota_df": "true", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_bytes": "0", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_max_periods": "4", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_readahead_min": "131072", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_reconnect_stale": "false", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_shutdown_timeout": "30", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_snapdir": ".snap", 2024-03-04T16:04:26.323 INFO:teuthology.orchestra.run.smithi038.stdout: "client_tick_interval": "1", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_trace": "", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_try_dentry_invalidate": "false", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_faked_inos": "false", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "client_use_random_mds": "false", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog": "false", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_graylog_port": "12201", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_monitors": "default=true", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog": "false", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "clog_to_syslog_level": "info", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "-", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network": "", 2024-03-04T16:04:26.324 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_network_interface": "", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_isal": "false", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_level": "5", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zlib_winsize": "-15", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "compressor_zstd_level": "1", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_op_num_threads": "6", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crimson_seastar_cpu_cores": "", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location": "", 2024-03-04T16:04:26.325 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook": "", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_location_hook_timeout": "10", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "daemonize": "false", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "daos_pool": "tank", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_alienstore": "0/5", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_allow_any_pool_priority": "false", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok": "1/5", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asok_assert_abort": "false", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_asserts_on_shutdown": "true", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_auth": "1/5", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bdev": "1/3", 2024-03-04T16:04:26.326 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluefs": "1/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_bluestore": "1/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_buffer": "0/1", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ceph_exporter": "1/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephfs_mirror": "0/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cephsqlite": "0/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_client": "0/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_compressor": "1/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_context": "0/1", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crimson_interrupt": "0/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crush": "1/1", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_crypto": "1/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_cyanstore": "0/5", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_deliberately_leak_memory": "false", 2024-03-04T16:04:26.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_disable_randomized_ping": "false", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_dpdk": "1/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_eventtrace": "1/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filer": "0/1", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_filestore": "1/3", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_finisher": "1/1", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_fuse": "1/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeat_testing_span": "0", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_heartbeatmap": "1/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_javaclient": "1/5", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journal": "1/3", 2024-03-04T16:04:26.328 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_journaler": "0/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_kstore": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_lockdep": "0/1", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mclock": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_balancer": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_locker": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_log_expire": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mds_migrator": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_memstore": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgr": "2/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mgrc": "1/5", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_mon": "20/20", 2024-03-04T16:04:26.329 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_monc": "0/10", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_ms": "1/1", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_none": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objclass": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objectcacher": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_objecter": "0/1", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_optracker": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_osd": "1/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_paxos": "20/20", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_perfcounter": "1/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_prioritycache": "1/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rados": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_mirror": "0/5", 2024-03-04T16:04:26.330 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_pwl": "0/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rbd_replay": "0/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_refs": "0/0", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_reserver": "1/1", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_access": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_datacache": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_dbstore": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_flight": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rgw_sync": "1/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_rocksdb": "4/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore": "0/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_backref": "0/5", 2024-03-04T16:04:26.331 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cache": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_cleaner": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_device": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_epm": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_journal": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_lba": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_odata": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_omap": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_onode": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_t": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_seastore_tm": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_striper": "0/1", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_test": "0/5", 2024-03-04T16:04:26.332 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_throttle": "1/1", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_timer": "0/1", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_tp": "0/5", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "debug_trace": "1/5", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "device_failure_prediction_mode": "none", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_graylog": "false", 2024-03-04T16:04:26.334 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_journald": "false", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_stderr": "true", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "err_to_syslog": "false", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event_tracing": "false", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_addr": "0.0.0.0", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_http_port": "9926", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_prio_limit": "5", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_sort_metrics": "true", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "exporter_stats_period": "5", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "fake_statfs_for_testing": "0", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "fatal_signal_handlers": "true", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_purge_ops": "10", 2024-03-04T16:04:26.335 INFO:teuthology.orchestra.run.smithi038.stdout: "filer_max_truncate_ops": "128", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_apply_finisher_threads": "1", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_blackhole": "false", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_clone_range": "true", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_btrfs_snap": "true", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_caller_concurrency": "10", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_collect_device_partition_information": "true", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_commit_timeout": "600.000000", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_inject_read_err": "false", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_omap_check": "false", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_debug_verify_split": "false", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_dump_file": "", 2024-03-04T16:04:26.336 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fadvise": "true", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fail_eio": "true", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_shards": "16", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fd_cache_size": "128", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap": "false", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fiemap_threshold": "4096", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_inject_stall": "0", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_parallel": "false", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_trailing": "false", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_journal_writeahead": "false", 2024-03-04T16:04:26.337 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_kill_at": "0", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs": "0", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size": "0", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-04T16:04:26.338 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_merge_threshold": "-10", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_odsync_write": "false", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend": "rocksdb", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_backend_path": "", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_thread_timeout": "60", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_op_threads": "2", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_punch_hole": "false", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-04T16:04:26.339 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_queue_max_ops": "50", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_seek_data_hole": "false", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc": "false", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_splice": "false", 2024-03-04T16:04:26.340 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_multiple": "2", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_split_rand_factor": "20", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_update_to": "1000", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_enable": "true", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-04T16:04:26.341 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_xfs_extsize": "false", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "filestore_zfs_snap": "false", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fio_dir": "/tmp/fio", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_allow_other": "true", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_atomic_o_trunc": "true", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_big_writes": "true", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_debug": "false", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_default_permissions": "false", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_disable_pagecache": "false", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_max_write": "0", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_multithreaded": "true", 2024-03-04T16:04:26.342 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_require_active_mds": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_set_user_groups": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_move": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_read": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_splice_write": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "fuse_use_invalidate_cb": "true", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "gss_target_name": "ceph", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_file": "", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_inject_failure": "0", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_interval": "5", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "host": "smithi038", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-04T16:04:26.343 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "inject_early_sigterm": "false", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_agent_port": "6799", 2024-03-04T16:04:26.344 INFO:teuthology.orchestra.run.smithi038.stdout: "jaeger_tracing_enable": "false", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_aio": "true", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_align_min_size": "65536", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_align": "true", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_block_size": "4096", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_dio": "true", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_discard": "false", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_force_aio": "false", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_ignore_corruption": "false", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_bytes": "10485760", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_max_write_entries": "100", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_replay_from": "0", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-04T16:04:26.345 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_write_header_frequency": "0", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journal_zero_on_create": "false", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prefetch_periods": "10", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_prezero_periods": "5", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "journaler_write_head_interval": "15", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "key": "", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "keyfile": "", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_backend": "rocksdb", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_default_stripe_size": "65536", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount": "false", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_bytes": "67108864", 2024-03-04T16:04:26.346 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_max_ops": "512", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_nid_prealloc": "1024", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_onode_map_size": "1024", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_submit_transaction": "false", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "kstore_sync_transaction": "false", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "librados_thread_count": "2", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep": "false", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "lockdep_force_backtrace": "false", 2024-03-04T16:04:26.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_coarse_timestamps": "true", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_flush_on_exit": "false", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_host": "127.0.0.1", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_graylog_port": "12201", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_new": "1000", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_max_recent": "10000", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stderr_prefix": "", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_stop_at_utilization": "0.970000", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": "true", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_graylog": "false", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_journald": "false", 2024-03-04T16:04:26.348 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_stderr": "false", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_syslog": "false", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "max_rotating_auth_attempts": "10", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_action_on_write_error": "1", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_alternate_name_max": "8192", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_asio_thread_count": "2", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_export_pin": "true", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_dirs": "true", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_interval": "5", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_interval": "10", 2024-03-04T16:04:26.349 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max": "-1", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_max_until": "-1", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_merge_size": "50", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_midchunk": "0.300000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_min_start": "0.200000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_minchunk": "0.001000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_mode": "0", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_max": "1.200000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_need_min": "0.800000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_overload_epochs": "2", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_bits": "3", 2024-03-04T16:04:26.350 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_size": "10000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_target_decay": "10.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_grace": "15.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_interval": "4.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_mid": "0.700000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_release_free_interval": "10", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_reservation": "0.050000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_interval": "1", 2024-03-04T16:04:26.351 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cache_trim_threshold": "262144", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_prealloc_inos": "1000", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_connect_bootstrapping": "false", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_damage_table_max_entries": "10000", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_auth_pins": "false", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_frag": "false", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_scatterstat": "false", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_debug_subtrees": "false", 2024-03-04T16:04:26.352 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_decay_halflife": "5.000000", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_default_dir_hash": "2", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_defer_session_stale": "true", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_deny_all_reconnect": "false", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_keys_per_op": "16384", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_commit_size": "10", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_max_entries": "0", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dir_prefetch": "true", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_on_map": "false", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_early_reply": "true", 2024-03-04T16:04:26.353 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enable_op_tracker": "true", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_enforce_unique_name": "true", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random": "true", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-04T16:04:26.354 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_health_summarize_threshold": "10", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_health_dummy": "false", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_migrator_session_race": "false", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_join_fs": "", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_journal_format": "1", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_create_at": "0", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_export_at": "0", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_import_at": "0", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_at": "0", 2024-03-04T16:04:26.355 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_expire_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_journal_replay_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_link_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_mdstable_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_openc_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_rename_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_shutdown_at": "0", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_event_large_threshold": "524288", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_events_per_segment": "1024", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_events": "-1", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_max_segments": "128", 2024-03-04T16:04:26.356 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_pause": "false", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_segment_size": "0", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_threshold": "128", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_log_warn_factor": "2.000000", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_caps_per_client": "1048576", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_flushes": "100000", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_completed_requests": "100000", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_export_size": "20971520", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_file_recover": "32", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_files": "64", 2024-03-04T16:04:26.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops": "8192", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_max_snaps_per_dir": "100", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_metrics_update_interval": "2", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_per_client": "100", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_min_caps_working_set": "10000", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_numa_node": "-1", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_complaint_time": "30.000000", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_duration": "600", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_size": "20", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_size": "20", 2024-03-04T16:04:26.358 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_op_log_threshold": "5", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_grace": "15", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_ping_interval": "5", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_caps": "30000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_recall_warning_threshold": "262144", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_interval": "1.000000", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-04T16:04:26.359 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_gid": "0", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_root_ino_uid": "0", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_shutdown_check": "0", 2024-03-04T16:04:26.360 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_skip_ino": "0", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_max_uid": "4294967294", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_min_uid": "0", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_snap_rstat": "false", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_standby_replay_damaged": "false", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_symlink_recovery": "true", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_exports": "0", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_thrash_fragments": "0", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_tick_interval": "5.000000", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_valgrind_exit": "false", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_backtrace": "1", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_verify_scatter": "false", 2024-03-04T16:04:26.361 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_wipe_sessions": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool_debug": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_device_bytes": "1073741824", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_set": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "memstore_page_size": "65536", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_bytes": "134217728", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_messages": "512", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_disabled_modules": "", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-04T16:04:26.362 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_max_pg_num_change": "128", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_bytes": "134217728", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mds_messages": "128", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_bytes": "134217728", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_mon_messages": "128", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_bytes": "536870912", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_osd_messages": "8192", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_pool": "true", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_standby_modules": "true", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_period": "5", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_stats_threshold": "5", 2024-03-04T16:04:26.363 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_tick_period": "2", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_delete": "true", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_allow_pool_size_one": "false", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_auth_validate_all_caps": "true", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_bytes": "104857600", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_directed_command_retry": "2", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-04T16:04:26.364 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_hunt_parallel": "3", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_log_interval": "1.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_interval": "10.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_client_target_rank": "-1", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_file_level": "debug", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_file": "true", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-04T16:04:26.365 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_journald": "true", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_bootstrap": "false", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_start": "false", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_compact_on_trim": "true", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_cpu_threads": "4", 2024-03-04T16:04:26.366 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_crush_min_required_version": "hammer", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_daemon_bytes": "419430400", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_crit": "5", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_avail_warn": "5", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_data_size_warn": "16106127360", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_json": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_dump_transactions": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_extra_checks": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-04T16:04:26.367 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_reef": "false", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_no_require_squid": "false", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_down_mkfs_grace": "60", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_default_strategy": "1", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_election_timeout": "5.000000", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_divisor": "2", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_enable_op_tracker": "true", 2024-03-04T16:04:26.368 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_fake_pool_delete": "false", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_force_quorum_join": "false", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_globalid_prealloc": "10000", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_detail_to_clog": "true", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_log_update_period": "5", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_max_detail": "50", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog": "true", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_interval": "600", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host": "172.21.15.38:6789/0", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_host_override": "", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_initial_members": "", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-04T16:04:26.369 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease": "5.000000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_full_interval": "50", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max": "10000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_log_max_summary": "50", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_log_epochs": "500", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_osd": "10000", 2024-03-04T16:04:26.370 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pg_per_osd": "10000", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_pool_pg_num": "65536", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_force_trim_to": "0", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mds_skip_sanity": "false", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_autotune": "true", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_memory_target": "2147483648", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_beacon_grace": "30", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_digest_period": "5", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_inactive_grace": "60", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-04T16:04:26.371 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_min_osdmap_epochs": "50", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_complaint_time": "30", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_duration": "600", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_size": "20", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_size": "20", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_op_log_threshold": "5", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_in": "false", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-04T16:04:26.372 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size": "500", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_interval": "600", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_force_trim_to": "0", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-04T16:04:26.373 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_down_reporters": "2", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_report_timeout": "900", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-04T16:04:26.374 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_stuck_threshold": "60", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-04T16:04:26.375 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_probe_timeout": "2.000000", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_change": "0.050000", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_max_osds": "4", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_interval": "86400", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_max_keys": "100", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_scrub_timeout": "300", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_session_timeout": "300", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_smart_report_timeout": "5", 2024-03-04T16:04:26.376 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stat_smooth_intervals": "6", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_min_size": "2", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_pool_size": "4", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_debug": "false", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_provider_kill_at": "0", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_requester_kill_at": "0", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_sync_timeout": "60.000000", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_target_pg_per_osd": "100", 2024-03-04T16:04:26.377 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_tick_interval": "5", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_interval": "300.000000", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_use_min_delay_socket": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_older_version_delay": "604800", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_misplaced": "false", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-04T16:04:26.378 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_older_version": "true", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "monmap": "", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-04T16:04:26.379 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "motr_tracing_enabled": "false", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_op_threads": "3", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_cm": "false", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_device_name": "", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_dscp": "96", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_local_gid": "", 2024-03-04T16:04:26.380 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_port_num": "1", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_sl": "3", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_support_srq": "true", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_rdma_type": "ib", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_async_reap_threshold": "5", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_before_connect": "false", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv4": "true", 2024-03-04T16:04:26.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_ipv6": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr1": "true", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_msgr2": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_max": "7568", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_port_min": "6800", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_count": "3", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_bind_retry_delay": "5", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_client": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mds": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mgr": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_mon": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_blackhole_osd": "false", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_client_mode": "crc secure", 2024-03-04T16:04:26.382 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_mode": "crc secure", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_cluster_type": "", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_compress_secure": "false", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_idle_timeout": "900", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_connection_ready_timeout": "10", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_data": "true", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_crc_header": "true", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bad_msg": "false", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_bug": "true", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_old_message": "true", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_skipped_message": "false", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_coremask": "0xF", 2024-03-04T16:04:26.383 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_enable_tso": "true", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hugepages": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_lro": "true", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_memory_channel": "4", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_pmd": "", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_port_id": "0", 2024-03-04T16:04:26.384 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_dump_on_send": "false", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_initial_backoff": "0.200000", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_max": "1.000000", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_delay_type": "", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_network_congestion": "0", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_inject_socket_failures": "0", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_learn_addr_from_peer": "true", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_accept_failures": "4", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_max_backoff": "15.000000", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_client_mode": "secure crc", 2024-03-04T16:04:26.385 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_mon_service_mode": "secure crc", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_min_size": "1024", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compress_mode": "none", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_pq_min_cost": "65536", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_public_type": "", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_service_mode": "crc secure", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_listen_backlog": "512", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_nodelay": "true", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_tcp_rcvbuf": "0", 2024-03-04T16:04:26.386 INFO:teuthology.orchestra.run.smithi038.stdout: "ms_type": "async+posix", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "no_config_file": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_completion_locks_per_session": "32", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inflight_ops": "1024", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_tick_interval": "5.000000", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objecter_timeout": "10.000000", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_blackhole": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "openssl_engine_opts": "", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_delay_time": "5.000000", 2024-03-04T16:04:26.387 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_hist_halflife": "1000", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_low_ops": "2", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_max_ops": "4", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_agent_slop": "0.020000", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_asio_thread_count": "2", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_async_recovery_min_cost": "100", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_max": "512", 2024-03-04T16:04:26.388 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backfill_scan_min": "64", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_degraded": "false", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_peering": "false", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_backoff_on_unfound": "true", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_beacon_report_interval": "300", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_duration": "30", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_max_block_size": "67108864", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blkin_trace_all": "false", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-04T16:04:26.389 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_for_log_corruption": "false", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_class_update_on_start": "true", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_cap": "256", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_message_size_cap": "524288000", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_op_priority": "63", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_client_watch_timeout": "30", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_command_max_records": "256", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_compact_on_start": "false", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-04T16:04:26.390 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_on_start": "true", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_crush_update_weight_set": "true", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_feed_pullee": "-1", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_misdirected_ops": "false", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_acting_change": "false", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_no_purge_strays": "false", 2024-03-04T16:04:26.391 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_op_order": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_shutdown": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_trim_objects": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_snaps": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-04T16:04:26.392 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_keys": "1024", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_stride": "524288", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_default_notify_timeout": "30", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep": "0.000000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-04T16:04:26.393 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_discard_disconnected_ops": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_enable_op_tracker": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_info": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_function_tracing": "false", 2024-03-04T16:04:26.394 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_grace": "20", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_interval": "6", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_peers": "10", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_min_size": "2000", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_stale": "600", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_max_size": "100000", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_min_size": "1000", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-04T16:04:26.395 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_journal_size": "5120", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_kill_backfill_at": "0", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_cache_size": "50", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_dedup": "true", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max": "40", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_map_share_max_epochs": "40", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_name_len": "100", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_attr_size": "0", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_backfills": "1", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_count": "5", 2024-03-04T16:04:26.396 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_markdown_period": "600", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_name_len": "2048", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_namespace_len": "256", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_object_size": "134217728", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_blocked_by": "16", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_log_entries": "10000", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_pgls": "1024", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_cost": "8388608", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_push_objects": "10", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_scrubs": "3", 2024-03-04T16:04:26.397 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_trimming_pgs": "2", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_op_reply_len": "64", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_max_write_size": "90", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_profile": "balanced", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-04T16:04:26.398 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_base": "805306368", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_min": "134217728", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target": "4294967296", 2024-03-04T16:04:26.399 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_autotune": "false", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_pg_log_entries": "250", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_min_recovery_priority": "0", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_interval": "5", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_cache_shards": "32", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_num_op_tracker_shard": "32", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_auto_affinity": "true", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_node": "-1", 2024-03-04T16:04:26.400 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_numa_prefer_iface": "true", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objecter_finishers": "1", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore": "bluestore", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_fuse": "false", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_objectstore_tracing": "false", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_complaint_time": "30.000000", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_duration": "600", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_size": "20", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_size": "20", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_log_threshold": "5", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards": "0", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_hdd": "5", 2024-03-04T16:04:26.401 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_shards_ssd": "8", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_pq_min_cost": "65536", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_queue_cut_off": "high", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_op_thread_timeout": "15", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_open_classes_on_start": "true", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_os_flags": "0", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_peering_op_priority": "255", 2024-03-04T16:04:26.402 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_cost": "1048576", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_delete_priority": "5", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_max": "10000", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_log_trim_min": "100", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-04T16:04:26.403 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crimson": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_flags": "0", 2024-03-04T16:04:26.404 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_min_size": "0", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pg_num": "32", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_pgp_num": "0", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_read_ratio": "70", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_size": "3", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_default_type": "replicated", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_push_per_object_cost": "1000", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap": "true", 2024-03-04T16:04:26.405 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_cost": "20971520", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active": "0", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_max_single_start": "1", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_priority": "3", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_priority": "5", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-04T16:04:26.406 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep": "0.000000", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_repair_during_recovery": "false", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_requested_scrub_priority": "120", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair": "false", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_hour": "0", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_begin_week_day": "0", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_max": "25", 2024-03-04T16:04:26.407 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_chunk_min": "5", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_cost": "52428800", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_during_recovery": "false", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_hour": "0", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_end_week_day": "0", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_event_cost": "4096", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_invalid_stats": "true", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-04T16:04:26.408 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_max_preemptions": "5", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_priority": "5", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_sleep": "0.000000", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_skip_data_digest": "false", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_smart_report_timeout": "5", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_cost": "1048576", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_priority": "5", 2024-03-04T16:04:26.409 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_target_transaction_size": "30", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-04T16:04:26.410 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_tracing": "false", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_use_stale_snap": "false", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "osdc_blkin_trace_all": "false", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_kill_at": "0", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_max_join_drift": "10", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min": "500", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_min_wait": "0.050000", 2024-03-04T16:04:26.411 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_propose_interval": "1.000000", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max": "500", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_service_trim_min": "10", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_stash_full_interval": "25", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_max": "500", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "paxos_trim_min": "250", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "perf": "true", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "pid_file": "", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "-", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrv": "", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "public_bind_addr": "-", 2024-03-04T16:04:26.412 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network": "", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "public_network_interface": "", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_enabled": "false", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "qat_compressor_session_max_number": "256", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_mon_op_timeout": "0", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_osd_op_timeout": "0", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rados_tracing": "false", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_atime_update_interval": "60", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_parent_reads": "false", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_balance_snap_reads": "false", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blkin_trace_all": "false", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-04T16:04:26.413 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache": "true", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_policy": "writearound", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_size": "33554432", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_clone_copy_on_read": "false", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_compression_hint": "none", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_concurrent_management_ops": "10", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-04T16:04:26.414 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_clone_format": "auto", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_data_pool": "", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_features": "61", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_format": "2", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_map_options": "", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_order": "22", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_pool": "rbd", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_count": "0", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_default_stripe_unit": "0", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_enable_alloc_hint": "true", 2024-03-04T16:04:26.415 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler": "simple", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_order": "24", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_pool": "", 2024-03-04T16:04:26.416 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_journal_splay_width": "4", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_parent_reads": "false", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_localize_snap_reads": "false", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-04T16:04:26.417 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-04T16:04:26.418 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_mtime_update_interval": "60", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_non_blocking_aio": "true", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_thread_timeout": "60", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_op_threads": "1", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_parent_cache_enabled": "false", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-04T16:04:26.419 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_plugins": "", 2024-03-04T16:04:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst": "0", 2024-03-04T16:04:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-04T16:04:26.420 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_bps_limit": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_exclude_ops": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_iops_limit": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-04T16:04:26.421 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_read_from_replica_policy": "default", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_skip_partial_discard": "true", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-04T16:04:26.422 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_tracing": "false", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_names": "true", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_validate_pool": "true", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_base_url": "", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "restapi_log_level": "", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_acl_grants_max_num": "100", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_admin_entry": "admin", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_backend_store": "rados", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_barbican_url": "", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_beast_enable_async": "true", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache": "false", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-04T16:04:26.423 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_enabled": "true", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_expiry_interval": "900", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cache_lru_size": "10000", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_config_store": "rados", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_content_length_compat": "false", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress": "true", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-04T16:04:26.424 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cors_rules_max_num": "100", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_cross_domain_policy": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_addr": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_password": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_kmip_username": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_require_ssl": "true", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-04T16:04:26.425 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-04T16:04:26.426 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_addr": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_auth": "token", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_namespace": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_prefix": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_token_file": "", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_buffersize": "524288", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_low_speed_time": "300", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-04T16:04:26.427 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_d4n_port": "6379", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_changes_size": "1000", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_num_shards": "128", 2024-03-04T16:04:26.428 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_log_window": "30", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-04T16:04:26.429 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_name": "", 2024-03-04T16:04:26.430 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dns_s3website_name": "", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_dynamic_resharding": "true", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_gc_threads": "true", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_lc_threads": "true", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_ops_log": "false", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_quota_threads": "true", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_static_website": "false", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enable_usage_log": "false", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_enforce_swift_acls": "true", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_exit_timeout_secs": "120", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_expose_bucket": "false", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_extended_http_attrs": "", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_filter": "none", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-04T16:04:26.431 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_frontends": "beast port=7480", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred": "50", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_objs": "32", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_gc_processor_period": "3600", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-04T16:04:26.432 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_init_timeout": "300", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_connection_idle": "30", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_domain": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_password_path": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_project": "", 2024-03-04T16:04:26.433 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_tenant": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_token_path": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_admin_user": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_api_version": "2", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_domain": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_password": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_project": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_barbican_user": "", 2024-03-04T16:04:26.435 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_url": "", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_debug_interval": "-1", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_lock_max_time": "90", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_objs": "32", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_rules": "1000", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_worker": "3", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-04T16:04:26.436 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lc_thread_delay": "0", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_dnattr": "uid", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_searchfilter": "", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_http_headers": "", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_log_object_name_utc": "false", 2024-03-04T16:04:26.437 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_name_len": "0", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attr_size": "0", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_chunk_size": "4194304", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_listing_results": "1000", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_notify_retries": "10", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_param_size": "1048576", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_put_size": "5368709120", 2024-03-04T16:04:26.438 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_max_slo_entries": "1000", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_log_max_shards": "64", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_mp_lock_max_time": "600", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-04T16:04:26.439 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_max_gc": "300", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_async_rados_threads": "32", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_num_control_oids": "8", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_numa_node": "-1", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-04T16:04:26.440 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_chunk_size": "100", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_gc_interval": "600", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_thread_timeout": "600", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_op_tracing": "false", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_token": "", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_url": "", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_opa_verify_ssl": "true", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-04T16:04:26.441 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_rados": "false", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_ops_log_socket_path": "", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval": "2.000000", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lanes": "3", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-04T16:04:26.442 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_cache_partitions": "3", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_continue": "true", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rados_tracing": "false", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm": "", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_id": "", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region": "", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-04T16:04:26.443 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_request_uri": "", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_batch_size": "64", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_max_aio": "128", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_num_logs": "16", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_reshard_thread_interval": "600", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_resolve_cname": "false", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_run_sync_thread": "true", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-04T16:04:26.444 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_scheduler_type": "throttler", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_script_uri": "", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_service_provider_name": "", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_id": "", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_client_secret": "", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_entry": "sts", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_key": "sts", 2024-03-04T16:04:26.445 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_min_session_duration": "900", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sts_token_introspection_url": "", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_account_in_url": "false", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_entry": "auth", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_auth_url": "", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_custom_header": "", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_need_stats": "true", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_tenant_name": "", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_token_expiration": "86400", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url": "", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_url_prefix": "swift", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-04T16:04:26.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_lease_period": "120", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_thread_pool_size": "512", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-04T16:04:26.447 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_comment": "", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_createby": "", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_encoding": "", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_flag": "false", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_origin": "", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_torrent_tracker": "", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_trust_forwarded_https": "false", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-04T16:04:26.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_shards": "32", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_usage_max_user_shards": "1", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_use_opa_authz": "false", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache": "false", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_max_buckets": "1000", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_policies_max_num": "100", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_user_unique_email": "true", 2024-03-04T16:04:26.449 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_verify_ssl": "true", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone": "", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_id": "", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup": "", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_id": "", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_auto_session_clear": "true", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_block_size": "4096", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-04T16:04:26.450 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_size": "536870912", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_index_type": "binary_search", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_partition_filters": "false", 2024-03-04T16:04:26.451 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_perf": "false", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "run_dir": "/var/run/ceph", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_block_create": "true", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cache_lru_size": "67108864", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_max_object_size": "16777216", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_device_size": "53687091200", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_capacity": "16", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-04T16:04:26.452 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_main_device_type": "SSD", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "seastore_segment_size": "67108864", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "set_keepcaps": "false", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "setgroup": "ceph", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser": "ceph", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "setuser_match_path": "", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "thp": "false", 2024-03-04T16:04:26.453 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_default_timeout": "60", 2024-03-04T16:04:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-04T16:04:26.454 INFO:teuthology.orchestra.run.smithi038.stdout: "throttler_perf_counter": "true" 2024-03-04T16:04:26.454 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:04:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: 3.d deep-scrub starts 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: 3.d deep-scrub ok 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:26 smithi176 ceph-mon[33619]: Deploying daemon node-exporter.b on smithi176 2024-03-04T16:04:26.609 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:26 smithi176 bash[64215]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-04T16:04:26.619 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: 3.d deep-scrub starts 2024-03-04T16:04:26.619 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: 3.d deep-scrub ok 2024-03-04T16:04:26.619 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[29779]: Deploying daemon node-exporter.b on smithi176 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: 3.d deep-scrub starts 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: 3.d deep-scrub ok 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: pgmap v48: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 771 B/s rd, 0 op/s 2024-03-04T16:04:26.620 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:26.621 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:26 smithi038 ceph-mon[38113]: Deploying daemon node-exporter.b on smithi176 2024-03-04T16:04:27.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-04T16:04:27.356 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:27.668 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:27 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:04:27.668 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:27 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:04:27.669 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:27 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:04:27.669 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:27 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:04:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:27 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:04:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:27 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:04:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[29779]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:04:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:04:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[38113]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:04:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:28 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:04:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:28 smithi176 ceph-mon[33619]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:28 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:04:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:28 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:04:28.859 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:28 smithi176 bash[64215]: Getting image source signatures 2024-03-04T16:04:28.859 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:28 smithi176 bash[64215]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-04T16:04:28.859 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:28 smithi176 bash[64215]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-04T16:04:28.859 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:28 smithi176 bash[64215]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-04T16:04:29.135 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-03-04T16:04:29.148 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-03-04T16:04:29.157 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-03-04T16:04:29.158 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:04:29.158 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:04:29.158 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:04:29.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:04:29.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:04:29.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:04:29.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:04:29.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/658092271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:04:29.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:04:29.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:04:29.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:04:29.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:04:29.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:29 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/658092271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:04:29.804 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-04T16:04:29.816 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-04T16:04:29.819 INFO:tasks.radosbench:Beginning radosbench... 2024-03-04T16:04:29.819 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-04T16:04:29.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-04T16:04:29.820 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-04T16:04:29.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-04T16:04:29.821 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-04T16:04:29.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:29.821 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-04T16:04:29.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:04:29.822 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-04T16:04:29.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:04:29.822 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-04T16:04:29.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:04:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:29.855 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:29.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:29 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:04:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:29 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:04:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:29 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:04:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:29 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:04:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:29 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/658092271' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:04:29.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:29.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:30.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:30.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:30.135 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:30.144 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:30.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:30.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:30.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:30.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:30.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.497 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:30.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:30.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:30.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 6.13 deep-scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 6.13 deep-scrub ok 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 3.9 deep-scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[29779]: 3.9 deep-scrub ok 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 6.13 deep-scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 6.13 deep-scrub ok 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 3.9 deep-scrub starts 2024-03-04T16:04:30.561 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:30 smithi038 ceph-mon[38113]: 3.9 deep-scrub ok 2024-03-04T16:04:30.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:30.623 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:30.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:30.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:30.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 6.13 deep-scrub starts 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 6.13 deep-scrub ok 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 3.9 deep-scrub starts 2024-03-04T16:04:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:30 smithi176 ceph-mon[33619]: 3.9 deep-scrub ok 2024-03-04T16:04:30.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:30.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:30.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:30.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:30.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:31.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:31.126 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:31.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:31.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:31.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:31.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:31.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:31.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:04:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:04:31.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:31 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:04:31.504 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:31.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:31.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:31.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:31.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:31.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: osdmap e94: 8 total, 8 up, 8 in 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:04:31.815 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:31 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:04:31.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:31.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:31.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:31.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:32.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:32.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:32.108 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:31 smithi176 bash[64215]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-04T16:04:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.131 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:32.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:32.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.257 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:32.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:32.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:32.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:32.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:32.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:32.571 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:32.571 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:32.608 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 bash[64215]: Writing manifest to image destination 2024-03-04T16:04:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:04:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:04:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-mon[33619]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-04T16:04:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-mon[33619]: 5.14 deep-scrub starts 2024-03-04T16:04:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-mon[33619]: 5.14 deep-scrub ok 2024-03-04T16:04:32.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:32.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:32.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:32.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:04:32.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:04:32.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[29779]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-04T16:04:32.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[29779]: 5.14 deep-scrub starts 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[29779]: 5.14 deep-scrub ok 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[38113]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[38113]: 5.14 deep-scrub starts 2024-03-04T16:04:32.693 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:32 smithi038 ceph-mon[38113]: 5.14 deep-scrub ok 2024-03-04T16:04:32.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:32.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:32.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:32.886 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:32.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:32.935 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 podman[64215]: 2024-03-04 16:04:32.617461355 +0000 UTC m=+6.221291801 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-04T16:04:32.935 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 podman[64215]: 2024-03-04 16:04:32.716560551 +0000 UTC m=+6.320391027 container create e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:04:32.935 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 podman[64215]: 2024-03-04 16:04:32.935639844 +0000 UTC m=+6.539470278 container init e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:04:32.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:33.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:33.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:33.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:33.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.264 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:33.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 podman[64215]: 2024-03-04 16:04:32.938539369 +0000 UTC m=+6.542369814 container start e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.942Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.942Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.942Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.942Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-04T16:04:33.359 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=arp 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-04T16:04:33.360 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=edac 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-04T16:04:33.361 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=os 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-04T16:04:33.362 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=stat 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=time 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=uname 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.943Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.944Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b[64452]: ts=2024-03-04T16:04:32.944Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:33 smithi176 bash[64215]: e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca 2024-03-04T16:04:33.363 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:04:33 smithi176 systemd[1]: Started Ceph node-exporter.b for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:04:33.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:33.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:33.440 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:33.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:33 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:33.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.514 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:33.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:33.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:33.638 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:33.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:33.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:33.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:04:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:33 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:04:33.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:33.892 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:33.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:33.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.019 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:34.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:34.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:34.145 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:34.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:34.272 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:34.374 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:34.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:34.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:34.522 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:34.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:34 smithi038 ceph-mon[29779]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:34.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:34 smithi038 ceph-mon[38113]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:34.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.649 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:34.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:34 smithi176 ceph-mon[33619]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:34.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:34.899 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:34.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:35.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.025 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:35.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:35.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:35.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:35.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:35.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.278 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:35.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:35.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:35.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:35.471 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_0' created 2024-03-04T16:04:35.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:35.529 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:35.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:35.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:04:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:04:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/841745865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[29779]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:04:35.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:04:35.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/841745865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:35 smithi038 ceph-mon[38113]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:35.778 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:35.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:35 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:04:35.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:35 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:04:35.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/841745865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:35 smithi176 ceph-mon[33619]: from='client.34694 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-04T16:04:35.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:35.903 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:35.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:36.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:36.029 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:36.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:36.155 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:36.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:36.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:36.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-04T16:04:36.383 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:36.407 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:36.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:36.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:36.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[38113]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:36.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[38113]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-04T16:04:36.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[38113]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:04:36.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:36.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:36.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[29779]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:36.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[29779]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-04T16:04:36.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[29779]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:04:36.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:36.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:36 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:36.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.661 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:36.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:36.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:36 smithi176 ceph-mon[33619]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-04T16:04:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:36 smithi176 ceph-mon[33619]: from='client.34694 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-04T16:04:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:36 smithi176 ceph-mon[33619]: osdmap e95: 8 total, 8 up, 8 in 2024-03-04T16:04:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:36 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:36 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:36.889 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:36.913 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:36.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:37.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.040 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:37.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:37.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:37.164 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:37.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:37.290 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:37.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:37.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:37.486 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:04:37.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:37.543 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:37.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:37.668 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2692220619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2472728899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[29779]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2692220619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2472728899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:37 smithi038 ceph-mon[38113]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.770 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:37.793 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: osdmap e96: 8 total, 8 up, 8 in 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2692220619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2472728899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:37 smithi176 ceph-mon[33619]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:04:37.873 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:04:37.874 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:04:37.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:04:37.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:04:37.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:04:37.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:04:37.875 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:04:37.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:37.918 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:37.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:38.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:38.043 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:38.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:38.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:38.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:38.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:38.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:38.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.544 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:38.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:38.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:04:38.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:04:38.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:04:38.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:38.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:04:38.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:04:38.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:38.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-04T16:04:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:04:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:04:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: osdmap e97: 8 total, 8 up, 8 in 2024-03-04T16:04:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:38.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:38.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:38.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:38.968 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:04:38.968 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:04:39.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:39.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.050 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:39.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:39.151 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:39.175 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:39.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:39.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:39.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:39.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.432 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:39.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:39.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:39.559 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:39.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:39.685 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:39.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:39.810 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:39.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:39.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:39.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:40.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:40.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:40.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:04:40.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:40.199 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:40.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.327 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:40.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:40.436 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:04:40.436 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-04T16:04:40.436 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:04:40.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:40.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:04:40.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:40.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:40.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:40.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:40.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:40.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:40.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:40.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:40.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:41.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:41.087 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:41.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:41.217 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:41 smithi038 ceph-mon[29779]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:41 smithi038 ceph-mon[38113]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:41.280 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:41.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:41.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:41 smithi176 ceph-mon[33619]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:41.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:41.473 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:41.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:41.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:41.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:41.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:41.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:41.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:41.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:41.983 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:41.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:42.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:42.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:42.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:42.241 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:42.343 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:42.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:42.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:42.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:42.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:42.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:42.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:42.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:42.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:42.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:42.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:42.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:42.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:04:42.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:43.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:43.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:43 smithi176 ceph-mon[33619]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:43.364 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:43.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:43 smithi038 ceph-mon[29779]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:43.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:43 smithi038 ceph-mon[38113]: pgmap v60: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:04:43.376 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:43.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:43.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.518 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:43.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.648 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:43.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:43.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:43.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:43.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:43.898 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:43.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:43.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:44.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:44.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:44.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:44.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:44.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:44 smithi038 ceph-mon[29779]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:04:44.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:44.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:44 smithi038 ceph-mon[38113]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:04:44.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:44.279 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:44.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:44.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:44.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.540 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:44.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:44.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:44.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:44 smithi176 ceph-mon[33619]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:04:44.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:44.675 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:44.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.812 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:44.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:44.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:44.943 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:44.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:45.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:45.073 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:45.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:45.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:45.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[29779]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:04:45.163 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:45.163 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:45.163 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:45 smithi038 ceph-mon[38113]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:04:45.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.201 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:45.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:45.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:45.331 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:45.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:45.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:45.460 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:45.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:45.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:45 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:45 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:45 smithi176 ceph-mon[33619]: osdmap e98: 8 total, 8 up, 8 in 2024-03-04T16:04:45.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:45.711 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:45.733 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:45.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:45.841 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:45.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:45.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:45.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:46.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:46.098 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:46.133 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-04T16:04:46.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:46.222 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:04:46.222 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:04:46.228 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:46.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:46.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:46.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:46.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[29779]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/540525910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[29779]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[38113]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/540525910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:46 smithi038 ceph-mon[38113]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:46 smithi176 ceph-mon[33619]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:04:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/540525910' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:46 smithi176 ceph-mon[33619]: from='client.34712 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:04:46.609 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:46.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:04:46.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:04:46.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:46.739 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:46.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:46.865 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:46.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:46.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:46.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:47.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:47.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:47.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:47.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:47.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:47.374 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:47.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:47.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:47.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:47 smithi038 ceph-mon[29779]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:04:47.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:47 smithi038 ceph-mon[29779]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:04:47.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:47 smithi038 ceph-mon[38113]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:04:47.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:47 smithi038 ceph-mon[38113]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:04:47.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:47.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:04:47.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:47.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:47 smithi176 ceph-mon[33619]: from='client.34712 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:04:47.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:47 smithi176 ceph-mon[33619]: osdmap e99: 8 total, 8 up, 8 in 2024-03-04T16:04:47.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:47.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:47.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:47.858 DEBUG:teuthology.orchestra.run.smithi038:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-04T16:04:47.859 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-04T16:04:47.859 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-04T16:04:47.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:04:47.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:47.861 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-04T16:04:47.861 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-04T16:04:47.887 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:47.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:47.928 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:hints = 1 2024-03-04T16:04:47.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-04T16:04:47.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object prefix: benchmark_data_smithi038_77863 2024-03-04T16:04:47.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:04:47.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 0 0 0 0 0 0 - 0 2024-03-04T16:04:47.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.032 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:48.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:48.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:48.148 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:48 smithi038 ceph-mon[29779]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:48.161 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:48.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:48.302 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:48.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:48.437 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:48.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:48 smithi038 ceph-mon[38113]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:48.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:48.581 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:48 smithi176 ceph-mon[33619]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:48.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:48.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:48.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:48.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:48.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:48.928 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 1 15 3308 3293 205.8 205.812 0.0036182 0.00484277 2024-03-04T16:04:48.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:49.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:49.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:49.150 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:49.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.304 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:49.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:49.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:49.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:49 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:04:49.451 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:49 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:04:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:49.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:49.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:49.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:49 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:04:49.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:49.735 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:49.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:49.850 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:04:49.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:49.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:49.894 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:04:49.928 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 2 16 6202 6186 193.299 180.812 0.00415774 0.00515867 2024-03-04T16:04:49.939 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:04:49.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:49.987 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:04:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.013 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:50.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:50.033 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:04:50.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:04:50.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:50.115 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:04:50.145 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:50.151 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:04:50.158 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:04:50.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:50 smithi038 ceph-mon[29779]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:50.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/554492934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:50.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:50 smithi038 ceph-mon[38113]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:50.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/554492934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:50.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:50.290 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:50.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:50.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:50.433 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:50.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:50.566 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:50 smithi176 ceph-mon[33619]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:04:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/554492934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:04:50.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:50.676 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:50.696 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:50.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:50.828 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:50.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 3 15 8751 8736 181.986 159.375 0.00469109 0.00548503 2024-03-04T16:04:50.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:50.961 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:50.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:51.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:51.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:51.180 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:04:51.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:51.224 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:51.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:51.365 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:51.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:04:51.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:04:51.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:04:51.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:51.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:51.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:51.791 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:51.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:51.927 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:51.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:51.928 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 4 16 11342 11326 176.955 161.875 0.00473992 0.00564372 2024-03-04T16:04:52.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:52.061 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:52.070 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:52.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:52.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.222 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:52.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:52.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:52.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:52.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:52.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:04:52.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:04:52.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-03-04T16:04:52.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:52.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:52.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:52.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2136156534' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: osdmap e100: 8 total, 8 up, 8 in 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 712 B/s rd, 0 op/s 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:04:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:04:52.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:52.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:52.770 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:52.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:52.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:52.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 5 15 14029 14014 175.161 168 0.00502509 0.0057017 2024-03-04T16:04:52.932 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:52.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:53.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:53.077 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:53.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:53.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:53.311 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:04:53.344 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:04:53.446 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:53.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:53.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:53.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:53.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:53.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:04:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:04:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:04:53.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:04:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:53.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:53.715 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:53.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:53.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:04:53.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:53.921 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:53.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:53.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 6 16 16623 16607 172.976 162.062 0.00871655 0.00577513 2024-03-04T16:04:54.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:54.072 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:54.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:54.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:54.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:54.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:54.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:54 smithi038 ceph-mon[29779]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:04:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:54 smithi038 ceph-mon[29779]: pgmap v70: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-04T16:04:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:54 smithi038 ceph-mon[38113]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:04:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:54 smithi038 ceph-mon[38113]: pgmap v70: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-04T16:04:54.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:54.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:54 smithi176 ceph-mon[33619]: osdmap e101: 8 total, 8 up, 8 in 2024-03-04T16:04:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:54 smithi176 ceph-mon[33619]: pgmap v70: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-04T16:04:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:54.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:54.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:54.773 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:54.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:54.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:54.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 7 16 19055 19039 169.976 152 0.00785926 0.00587642 2024-03-04T16:04:54.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:54.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:55.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:55.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:55.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:55.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:55.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:04:55.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:55.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:55.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:55.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:55.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:55.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:55.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:55.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:55.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:55.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 8 16 21087 21071 164.603 127 0.00485837 0.0060683 2024-03-04T16:04:55.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:55.982 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:55.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:56.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:56.122 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:56.224 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:56.256 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:56.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:56.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:56.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[29779]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:04:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[29779]: pgmap v72: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-04T16:04:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[29779]: from='client.24739 v1:172.21.15.38:0/2708125834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[38113]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:04:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[38113]: pgmap v72: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-04T16:04:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:56 smithi038 ceph-mon[38113]: from='client.24739 v1:172.21.15.38:0/2708125834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:56.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:56 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:56 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:56 smithi176 ceph-mon[33619]: osdmap e102: 8 total, 8 up, 8 in 2024-03-04T16:04:56.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:56 smithi176 ceph-mon[33619]: pgmap v72: 148 pgs: 148 active+clean; 770 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 128 MiB/s wr, 2.05k op/s 2024-03-04T16:04:56.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:56 smithi176 ceph-mon[33619]: from='client.24739 v1:172.21.15.38:0/2708125834' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:04:56.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.671 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:56.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:56.772 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.796 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:56.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:56.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:56.929 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 9 16 23065 23049 160.048 123.625 0.004453 0.00624077 2024-03-04T16:04:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:56.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:56.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:57.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:57.074 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:57.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:57.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:57.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:57.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:57 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:04:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:57 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:04:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:57 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:04:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:57 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:04:57.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:57.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:57 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:04:57.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:57 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:04:57.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:57.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:57.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:04:57.753 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:04:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:57.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:04:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:04:57.868 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:57.905 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:57.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:57.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 10 16 24744 24728 154.529 104.938 0.00625431 0.00646528 2024-03-04T16:04:58.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.033 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:58.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:58.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:58.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:58.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:58.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:04:58.306 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:04:58.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:58.448 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:58.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:58 smithi176 ceph-mon[33619]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:04:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:04:58 smithi176 ceph-mon[33619]: pgmap v74: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 246 MiB/s wr, 3.93k op/s 2024-03-04T16:04:58.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:58.722 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:58 smithi038 ceph-mon[38113]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:04:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:04:58 smithi038 ceph-mon[38113]: pgmap v74: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 246 MiB/s wr, 3.93k op/s 2024-03-04T16:04:58.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:58 smithi038 ceph-mon[29779]: osdmap e103: 8 total, 8 up, 8 in 2024-03-04T16:04:58.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:04:58 smithi038 ceph-mon[29779]: pgmap v74: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 246 MiB/s wr, 3.93k op/s 2024-03-04T16:04:58.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:04:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:58.874 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:58.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:04:58.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 11 16 26574 26558 150.878 114.375 0.00325383 0.00662131 2024-03-04T16:04:58.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:59.010 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:59.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:59.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:59.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:04:59.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.269 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:59.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:59.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:04:59.408 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:04:59.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:59.554 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:59.565 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2024-03-04T16:04:55.302210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":807803230,"num_objects":12515,"num_object_clones":0,"num_object_copies":37545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12515,"num_whiteouts":0,"num_read":1057,"num_read_kb":903,"num_write":12932,"num_write_kb":790749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12945,"ondisk_log_size":12945,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2416920,"kb_used_data":2138364,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":747544296,"statfs":{"total":767960285184,"available":765485359104,"internally_reserved":0,"allocated":2189684736,"data_stored":2186623490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":285134386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,3,6,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":807206912,"num_objects":12317,"num_object_clones":0,"num_object_copies":36951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12317,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":12317,"num_write_kb":788288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001116"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171661+0000","last_change":"2024-03-04T16:03:57.477127+0000","last_active":"2024-03-04T16:04:51.171661+0000","last_peered":"2024-03-04T16:04:51.171661+0000","last_clean":"2024-03-04T16:04:51.171661+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:04:51.171661+0000","last_undegraded":"2024-03-04T16:04:51.171661+0000","last_fullsized":"2024-03-04T16:04:51.171661+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.477014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.477014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.118199+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183684+0000","last_change":"2024-03-04T16:04:28.004436+0000","last_active":"2024-03-04T16:04:51.183684+0000","last_peered":"2024-03-04T16:04:51.183684+0000","last_clean":"2024-03-04T16:04:51.183684+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:04:51.183684+0000","last_undegraded":"2024-03-04T16:04:51.183684+0000","last_fullsized":"2024-03-04T16:04:51.183684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:28.004355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:28.004355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:54.906901+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183401+0000","last_change":"2024-03-04T16:03:45.773330+0000","last_active":"2024-03-04T16:04:51.183401+0000","last_peered":"2024-03-04T16:04:51.183401+0000","last_clean":"2024-03-04T16:04:51.183401+0000","last_became_active":"2024-03-04T16:02:43.714846+0000","last_became_peered":"2024-03-04T16:02:43.714846+0000","last_unstale":"2024-03-04T16:04:51.183401+0000","last_undegraded":"2024-03-04T16:04:51.183401+0000","last_fullsized":"2024-03-04T16:04:51.183401+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:45.773213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.773213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.488104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171536+0000","last_change":"2024-03-04T16:04:15.666765+0000","last_active":"2024-03-04T16:04:51.171536+0000","last_peered":"2024-03-04T16:04:51.171536+0000","last_clean":"2024-03-04T16:04:51.171536+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:04:51.171536+0000","last_undegraded":"2024-03-04T16:04:51.171536+0000","last_fullsized":"2024-03-04T16:04:51.171536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:15.666674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.666674+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.309413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176862+0000","last_change":"2024-03-04T16:04:05.380482+0000","last_active":"2024-03-04T16:04:51.176862+0000","last_peered":"2024-03-04T16:04:51.176862+0000","last_clean":"2024-03-04T16:04:51.176862+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:04:51.176862+0000","last_undegraded":"2024-03-04T16:04:51.176862+0000","last_fullsized":"2024-03-04T16:04:51.176862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.380364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.380364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.798710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171644+0000","last_change":"2024-03-04T16:04:08.519836+0000","last_active":"2024-03-04T16:04:51.171644+0000","last_peered":"2024-03-04T16:04:51.171644+0000","last_clean":"2024-03-04T16:04:51.171644+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:04:51.171644+0000","last_undegraded":"2024-03-04T16:04:51.171644+0000","last_fullsized":"2024-03-04T16:04:51.171644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.519713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.519713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.148055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":76,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178040+0000","last_change":"2024-03-04T16:02:43.712774+0000","last_active":"2024-03-04T16:04:51.178040+0000","last_peered":"2024-03-04T16:04:51.178040+0000","last_clean":"2024-03-04T16:04:51.178040+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:04:51.178040+0000","last_undegraded":"2024-03-04T16:04:51.178040+0000","last_fullsized":"2024-03-04T16:04:51.178040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:02:42.653707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178628+0000","last_change":"2024-03-04T16:03:50.379226+0000","last_active":"2024-03-04T16:04:51.178628+0000","last_peered":"2024-03-04T16:04:51.178628+0000","last_clean":"2024-03-04T16:04:51.178628+0000","last_became_active":"2024-03-04T16:02:45.749383+0000","last_became_peered":"2024-03-04T16:02:45.749383+0000","last_unstale":"2024-03-04T16:04:51.178628+0000","last_undegraded":"2024-03-04T16:04:51.178628+0000","last_fullsized":"2024-03-04T16:04:51.178628+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:50.379077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.379077+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.153393+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182052+0000","last_change":"2024-03-04T16:03:56.851189+0000","last_active":"2024-03-04T16:04:51.182052+0000","last_peered":"2024-03-04T16:04:51.182052+0000","last_clean":"2024-03-04T16:04:51.182052+0000","last_became_active":"2024-03-04T16:02:49.791142+0000","last_became_peered":"2024-03-04T16:02:49.791142+0000","last_unstale":"2024-03-04T16:04:51.182052+0000","last_undegraded":"2024-03-04T16:04:51.182052+0000","last_fullsized":"2024-03-04T16:04:51.182052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.851094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.851094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.117129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180236+0000","last_change":"2024-03-04T16:04:20.132795+0000","last_active":"2024-03-04T16:04:51.180236+0000","last_peered":"2024-03-04T16:04:51.180236+0000","last_clean":"2024-03-04T16:04:51.180236+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:04:51.180236+0000","last_undegraded":"2024-03-04T16:04:51.180236+0000","last_fullsized":"2024-03-04T16:04:51.180236+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:04:20.132710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.994917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180316+0000","last_change":"2024-03-04T16:03:51.290213+0000","last_active":"2024-03-04T16:04:51.180316+0000","last_peered":"2024-03-04T16:04:51.180316+0000","last_clean":"2024-03-04T16:04:51.180316+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:04:51.180316+0000","last_undegraded":"2024-03-04T16:04:51.180316+0000","last_fullsized":"2024-03-04T16:04:51.180316+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.290117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.290117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.885059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171866+0000","last_change":"2024-03-04T16:04:00.434148+0000","last_active":"2024-03-04T16:04:51.171866+0000","last_peered":"2024-03-04T16:04:51.171866+0000","last_clean":"2024-03-04T16:04:51.171866+0000","last_became_active":"2024-03-04T16:02:45.745532+0000","last_became_peered":"2024-03-04T16:02:45.745532+0000","last_unstale":"2024-03-04T16:04:51.171866+0000","last_undegraded":"2024-03-04T16:04:51.171866+0000","last_fullsized":"2024-03-04T16:04:51.171866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:00.434035+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.434035+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.035919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.179935+0000","last_change":"2024-03-04T16:04:02.311228+0000","last_active":"2024-03-04T16:04:51.179935+0000","last_peered":"2024-03-04T16:04:51.179935+0000","last_clean":"2024-03-04T16:04:51.179935+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:04:51.179935+0000","last_undegraded":"2024-03-04T16:04:51.179935+0000","last_fullsized":"2024-03-04T16:04:51.179935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:02.311103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:02.311103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:06.585200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183040+0000","last_change":"2024-03-04T16:04:08.905885+0000","last_active":"2024-03-04T16:04:51.183040+0000","last_peered":"2024-03-04T16:04:51.183040+0000","last_clean":"2024-03-04T16:04:51.183040+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:04:51.183040+0000","last_undegraded":"2024-03-04T16:04:51.183040+0000","last_fullsized":"2024-03-04T16:04:51.183040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.905768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.905768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.441912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180579+0000","last_change":"2024-03-04T16:04:00.292627+0000","last_active":"2024-03-04T16:04:51.180579+0000","last_peered":"2024-03-04T16:04:51.180579+0000","last_clean":"2024-03-04T16:04:51.180579+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:04:51.180579+0000","last_undegraded":"2024-03-04T16:04:51.180579+0000","last_fullsized":"2024-03-04T16:04:51.180579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:00.292499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:00.292499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.161517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"75'5","reported_seq":101,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172410+0000","last_change":"2024-03-04T16:04:26.658074+0000","last_active":"2024-03-04T16:04:51.172410+0000","last_peered":"2024-03-04T16:04:51.172410+0000","last_clean":"2024-03-04T16:04:51.172410+0000","last_became_active":"2024-03-04T16:04:22.732082+0000","last_became_peered":"2024-03-04T16:04:22.732082+0000","last_unstale":"2024-03-04T16:04:51.172410+0000","last_undegraded":"2024-03-04T16:04:51.172410+0000","last_fullsized":"2024-03-04T16:04:51.172410+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-04T16:04:26.657973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:26.657973+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:50.292812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.173013+0000","last_change":"2024-03-04T16:04:27.630590+0000","last_active":"2024-03-04T16:04:51.173013+0000","last_peered":"2024-03-04T16:04:51.173013+0000","last_clean":"2024-03-04T16:04:51.173013+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:04:51.173013+0000","last_undegraded":"2024-03-04T16:04:51.173013+0000","last_fullsized":"2024-03-04T16:04:51.173013+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:27.630484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:27.630484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:48.437589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183839+0000","last_change":"2024-03-04T16:04:08.039737+0000","last_active":"2024-03-04T16:04:51.183839+0000","last_peered":"2024-03-04T16:04:51.183839+0000","last_clean":"2024-03-04T16:04:51.183839+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:04:51.183839+0000","last_undegraded":"2024-03-04T16:04:51.183839+0000","last_fullsized":"2024-03-04T16:04:51.183839+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.039649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.039649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.929286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183009+0000","last_change":"2024-03-04T16:04:12.724442+0000","last_active":"2024-03-04T16:04:51.183009+0000","last_peered":"2024-03-04T16:04:51.183009+0000","last_clean":"2024-03-04T16:04:51.183009+0000","last_became_active":"2024-03-04T16:02:43.714419+0000","last_became_peered":"2024-03-04T16:02:43.714419+0000","last_unstale":"2024-03-04T16:04:51.183009+0000","last_undegraded":"2024-03-04T16:04:51.183009+0000","last_fullsized":"2024-03-04T16:04:51.183009+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:12.724360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:12.724360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:18.288024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177707+0000","last_change":"2024-03-04T16:03:45.980725+0000","last_active":"2024-03-04T16:04:51.177707+0000","last_peered":"2024-03-04T16:04:51.177707+0000","last_clean":"2024-03-04T16:04:51.177707+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:04:51.177707+0000","last_undegraded":"2024-03-04T16:04:51.177707+0000","last_fullsized":"2024-03-04T16:04:51.177707+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:45.980606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:45.980606+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:01.702710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.179677+0000","last_change":"2024-03-04T16:04:21.388358+0000","last_active":"2024-03-04T16:04:51.179677+0000","last_peered":"2024-03-04T16:04:51.179677+0000","last_clean":"2024-03-04T16:04:51.179677+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:04:51.179677+0000","last_undegraded":"2024-03-04T16:04:51.179677+0000","last_fullsized":"2024-03-04T16:04:51.179677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:21.388278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:21.388278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.667572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172916+0000","last_change":"2024-03-04T16:04:16.650776+0000","last_active":"2024-03-04T16:04:51.172916+0000","last_peered":"2024-03-04T16:04:51.172916+0000","last_clean":"2024-03-04T16:04:51.172916+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:04:51.172916+0000","last_undegraded":"2024-03-04T16:04:51.172916+0000","last_fullsized":"2024-03-04T16:04:51.172916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.650697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.650697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.898928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171878+0000","last_change":"2024-03-04T16:03:46.423866+0000","last_active":"2024-03-04T16:04:51.171878+0000","last_peered":"2024-03-04T16:04:51.171878+0000","last_clean":"2024-03-04T16:04:51.171878+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:04:51.171878+0000","last_undegraded":"2024-03-04T16:04:51.171878+0000","last_fullsized":"2024-03-04T16:04:51.171878+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.423767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.423767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.868075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":102,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176891+0000","last_change":"2024-03-04T16:03:55.411658+0000","last_active":"2024-03-04T16:04:51.176891+0000","last_peered":"2024-03-04T16:04:51.176891+0000","last_clean":"2024-03-04T16:04:51.176891+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:04:51.176891+0000","last_undegraded":"2024-03-04T16:04:51.176891+0000","last_fullsized":"2024-03-04T16:04:51.176891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:55.411554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.411554+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.742258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":73,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183486+0000","last_change":"2024-03-04T16:04:17.871081+0000","last_active":"2024-03-04T16:04:51.183486+0000","last_peered":"2024-03-04T16:04:51.183486+0000","last_clean":"2024-03-04T16:04:51.183486+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:04:51.183486+0000","last_undegraded":"2024-03-04T16:04:51.183486+0000","last_fullsized":"2024-03-04T16:04:51.183486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:04:17.870988+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:46.277891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"75'8","reported_seq":113,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183483+0000","last_change":"2024-03-04T16:04:09.707358+0000","last_active":"2024-03-04T16:04:51.183483+0000","last_peered":"2024-03-04T16:04:51.183483+0000","last_clean":"2024-03-04T16:04:51.183483+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:04:51.183483+0000","last_undegraded":"2024-03-04T16:04:51.183483+0000","last_fullsized":"2024-03-04T16:04:51.183483+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.707234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.707234+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.027920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183775+0000","last_change":"2024-03-04T16:04:01.015009+0000","last_active":"2024-03-04T16:04:51.183775+0000","last_peered":"2024-03-04T16:04:51.183775+0000","last_clean":"2024-03-04T16:04:51.183775+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:04:51.183775+0000","last_undegraded":"2024-03-04T16:04:51.183775+0000","last_fullsized":"2024-03-04T16:04:51.183775+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:01.014871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:01.014871+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:13.772804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183668+0000","last_change":"2024-03-04T16:04:30.616496+0000","last_active":"2024-03-04T16:04:51.183668+0000","last_peered":"2024-03-04T16:04:51.183668+0000","last_clean":"2024-03-04T16:04:51.183668+0000","last_became_active":"2024-03-04T16:02:45.744705+0000","last_became_peered":"2024-03-04T16:02:45.744705+0000","last_unstale":"2024-03-04T16:04:51.183668+0000","last_undegraded":"2024-03-04T16:04:51.183668+0000","last_fullsized":"2024-03-04T16:04:51.183668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:04:30.616397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:30.616397+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:53.437591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184048+0000","last_change":"2024-03-04T16:03:53.955279+0000","last_active":"2024-03-04T16:04:51.184048+0000","last_peered":"2024-03-04T16:04:51.184048+0000","last_clean":"2024-03-04T16:04:51.184048+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:04:51.184048+0000","last_undegraded":"2024-03-04T16:04:51.184048+0000","last_fullsized":"2024-03-04T16:04:51.184048+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:53.955186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:53.955186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.764558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":76,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184041+0000","last_change":"2024-03-04T16:03:57.874266+0000","last_active":"2024-03-04T16:04:51.184041+0000","last_peered":"2024-03-04T16:04:51.184041+0000","last_clean":"2024-03-04T16:04:51.184041+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:04:51.184041+0000","last_undegraded":"2024-03-04T16:04:51.184041+0000","last_fullsized":"2024-03-04T16:04:51.184041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.874150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.874150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.705755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176407+0000","last_change":"2024-03-04T16:03:47.354437+0000","last_active":"2024-03-04T16:04:51.176407+0000","last_peered":"2024-03-04T16:04:51.176407+0000","last_clean":"2024-03-04T16:04:51.176407+0000","last_became_active":"2024-03-04T16:02:43.718570+0000","last_became_peered":"2024-03-04T16:02:43.718570+0000","last_unstale":"2024-03-04T16:04:51.176407+0000","last_undegraded":"2024-03-04T16:04:51.176407+0000","last_fullsized":"2024-03-04T16:04:51.176407+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.354324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.354324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:57.540186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":117,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183320+0000","last_change":"2024-03-04T16:04:22.740687+0000","last_active":"2024-03-04T16:04:51.183320+0000","last_peered":"2024-03-04T16:04:51.183320+0000","last_clean":"2024-03-04T16:04:51.183320+0000","last_became_active":"2024-03-04T16:04:22.734590+0000","last_became_peered":"2024-03-04T16:04:22.734590+0000","last_unstale":"2024-03-04T16:04:51.183320+0000","last_undegraded":"2024-03-04T16:04:51.183320+0000","last_fullsized":"2024-03-04T16:04:51.183320+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:51.720518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.720518+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:53.208762+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183035+0000","last_change":"2024-03-04T16:03:57.893125+0000","last_active":"2024-03-04T16:04:51.183035+0000","last_peered":"2024-03-04T16:04:51.183035+0000","last_clean":"2024-03-04T16:04:51.183035+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:04:51.183035+0000","last_undegraded":"2024-03-04T16:04:51.183035+0000","last_fullsized":"2024-03-04T16:04:51.183035+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.893007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.893007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.709209+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181859+0000","last_change":"2024-03-04T16:04:05.798912+0000","last_active":"2024-03-04T16:04:51.181859+0000","last_peered":"2024-03-04T16:04:51.181859+0000","last_clean":"2024-03-04T16:04:51.181859+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:04:51.181859+0000","last_undegraded":"2024-03-04T16:04:51.181859+0000","last_fullsized":"2024-03-04T16:04:51.181859+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:05.798829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.798829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.585226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.186921+0000","last_change":"2024-03-04T16:04:08.761418+0000","last_active":"2024-03-04T16:04:51.186921+0000","last_peered":"2024-03-04T16:04:51.186921+0000","last_clean":"2024-03-04T16:04:51.186921+0000","last_became_active":"2024-03-04T16:02:43.716542+0000","last_became_peered":"2024-03-04T16:02:43.716542+0000","last_unstale":"2024-03-04T16:04:51.186921+0000","last_undegraded":"2024-03-04T16:04:51.186921+0000","last_fullsized":"2024-03-04T16:04:51.186921+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:08.761341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:08.761341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:10.153431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":85,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177212+0000","last_change":"2024-03-04T16:04:20.386746+0000","last_active":"2024-03-04T16:04:51.177212+0000","last_peered":"2024-03-04T16:04:51.177212+0000","last_clean":"2024-03-04T16:04:51.177212+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:04:51.177212+0000","last_undegraded":"2024-03-04T16:04:51.177212+0000","last_fullsized":"2024-03-04T16:04:51.177212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:04:20.386619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:20.386619+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:21.336751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178631+0000","last_change":"2024-03-04T16:04:29.356938+0000","last_active":"2024-03-04T16:04:51.178631+0000","last_peered":"2024-03-04T16:04:51.178631+0000","last_clean":"2024-03-04T16:04:51.178631+0000","last_became_active":"2024-03-04T16:02:49.791216+0000","last_became_peered":"2024-03-04T16:02:49.791216+0000","last_unstale":"2024-03-04T16:04:51.178631+0000","last_undegraded":"2024-03-04T16:04:51.178631+0000","last_fullsized":"2024-03-04T16:04:51.178631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:04:29.356853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:54.403774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182792+0000","last_change":"2024-03-04T16:04:31.844715+0000","last_active":"2024-03-04T16:04:51.182792+0000","last_peered":"2024-03-04T16:04:51.182792+0000","last_clean":"2024-03-04T16:04:51.182792+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:04:51.182792+0000","last_undegraded":"2024-03-04T16:04:51.182792+0000","last_fullsized":"2024-03-04T16:04:51.182792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:31.844639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:31.844639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.731464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180658+0000","last_change":"2024-03-04T16:03:52.330168+0000","last_active":"2024-03-04T16:04:51.180658+0000","last_peered":"2024-03-04T16:04:51.180658+0000","last_clean":"2024-03-04T16:04:51.180658+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:04:51.180658+0000","last_undegraded":"2024-03-04T16:04:51.180658+0000","last_fullsized":"2024-03-04T16:04:51.180658+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.330026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.330026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:52.392092+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178523+0000","last_change":"2024-03-04T16:04:13.431846+0000","last_active":"2024-03-04T16:04:51.178523+0000","last_peered":"2024-03-04T16:04:51.178523+0000","last_clean":"2024-03-04T16:04:51.178523+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:04:51.178523+0000","last_undegraded":"2024-03-04T16:04:51.178523+0000","last_fullsized":"2024-03-04T16:04:51.178523+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:04:13.431752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.431752+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.426748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180873+0000","last_change":"2024-03-04T16:04:03.791013+0000","last_active":"2024-03-04T16:04:51.180873+0000","last_peered":"2024-03-04T16:04:51.180873+0000","last_clean":"2024-03-04T16:04:51.180873+0000","last_became_active":"2024-03-04T16:02:49.794197+0000","last_became_peered":"2024-03-04T16:02:49.794197+0000","last_unstale":"2024-03-04T16:04:51.180873+0000","last_undegraded":"2024-03-04T16:04:51.180873+0000","last_fullsized":"2024-03-04T16:04:51.180873+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.790905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.790905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.304942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":67,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176762+0000","last_change":"2024-03-04T16:02:47.777366+0000","last_active":"2024-03-04T16:04:51.176762+0000","last_peered":"2024-03-04T16:04:51.176762+0000","last_clean":"2024-03-04T16:04:51.176762+0000","last_became_active":"2024-03-04T16:02:47.777008+0000","last_became_peered":"2024-03-04T16:02:47.777008+0000","last_unstale":"2024-03-04T16:04:51.176762+0000","last_undegraded":"2024-03-04T16:04:51.176762+0000","last_fullsized":"2024-03-04T16:04:51.176762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183810+0000","last_change":"2024-03-04T16:04:23.876349+0000","last_active":"2024-03-04T16:04:51.183810+0000","last_peered":"2024-03-04T16:04:51.183810+0000","last_clean":"2024-03-04T16:04:51.183810+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:04:51.183810+0000","last_undegraded":"2024-03-04T16:04:51.183810+0000","last_fullsized":"2024-03-04T16:04:51.183810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:23.876232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:23.876232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:51.505700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":99,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185865+0000","last_change":"2024-03-04T16:04:07.040237+0000","last_active":"2024-03-04T16:04:51.185865+0000","last_peered":"2024-03-04T16:04:51.185865+0000","last_clean":"2024-03-04T16:04:51.185865+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:04:51.185865+0000","last_undegraded":"2024-03-04T16:04:51.185865+0000","last_fullsized":"2024-03-04T16:04:51.185865+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.040136+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.386047+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176399+0000","last_change":"2024-03-04T16:02:49.792155+0000","last_active":"2024-03-04T16:04:51.176399+0000","last_peered":"2024-03-04T16:04:51.176399+0000","last_clean":"2024-03-04T16:04:51.176399+0000","last_became_active":"2024-03-04T16:02:49.791815+0000","last_became_peered":"2024-03-04T16:02:49.791815+0000","last_unstale":"2024-03-04T16:04:51.176399+0000","last_undegraded":"2024-03-04T16:04:51.176399+0000","last_fullsized":"2024-03-04T16:04:51.176399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185603+0000","last_change":"2024-03-04T16:03:51.939795+0000","last_active":"2024-03-04T16:04:51.185603+0000","last_peered":"2024-03-04T16:04:51.185603+0000","last_clean":"2024-03-04T16:04:51.185603+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:04:51.185603+0000","last_undegraded":"2024-03-04T16:04:51.185603+0000","last_fullsized":"2024-03-04T16:04:51.185603+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.939712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.939712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.441874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172638+0000","last_change":"2024-03-04T16:03:44.419164+0000","last_active":"2024-03-04T16:04:51.172638+0000","last_peered":"2024-03-04T16:04:51.172638+0000","last_clean":"2024-03-04T16:04:51.172638+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:04:51.172638+0000","last_undegraded":"2024-03-04T16:04:51.172638+0000","last_fullsized":"2024-03-04T16:04:51.172638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:44.419065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:44.419065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.867222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172359+0000","last_change":"2024-03-04T16:03:50.400887+0000","last_active":"2024-03-04T16:04:51.172359+0000","last_peered":"2024-03-04T16:04:51.172359+0000","last_clean":"2024-03-04T16:04:51.172359+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:04:51.172359+0000","last_undegraded":"2024-03-04T16:04:51.172359+0000","last_fullsized":"2024-03-04T16:04:51.172359+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:03:50.400784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.400784+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:58.749407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183105+0000","last_change":"2024-03-04T16:02:49.797503+0000","last_active":"2024-03-04T16:04:51.183105+0000","last_peered":"2024-03-04T16:04:51.183105+0000","last_clean":"2024-03-04T16:04:51.183105+0000","last_became_active":"2024-03-04T16:02:49.797174+0000","last_became_peered":"2024-03-04T16:02:49.797174+0000","last_unstale":"2024-03-04T16:04:51.183105+0000","last_undegraded":"2024-03-04T16:04:51.183105+0000","last_fullsized":"2024-03-04T16:04:51.183105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:53.612269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"75'8","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180375+0000","last_change":"2024-03-04T16:04:10.217159+0000","last_active":"2024-03-04T16:04:51.180375+0000","last_peered":"2024-03-04T16:04:51.180375+0000","last_clean":"2024-03-04T16:04:51.180375+0000","last_became_active":"2024-03-04T16:02:47.768874+0000","last_became_peered":"2024-03-04T16:02:47.768874+0000","last_unstale":"2024-03-04T16:04:51.180375+0000","last_undegraded":"2024-03-04T16:04:51.180375+0000","last_fullsized":"2024-03-04T16:04:51.180375+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:10.217024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.217024+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:39.681485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":89,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183032+0000","last_change":"2024-03-04T16:04:06.837374+0000","last_active":"2024-03-04T16:04:51.183032+0000","last_peered":"2024-03-04T16:04:51.183032+0000","last_clean":"2024-03-04T16:04:51.183032+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:04:51.183032+0000","last_undegraded":"2024-03-04T16:04:51.183032+0000","last_fullsized":"2024-03-04T16:04:51.183032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:06.837263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.837263+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:25.977815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181603+0000","last_change":"2024-03-04T16:04:22.739538+0000","last_active":"2024-03-04T16:04:51.181603+0000","last_peered":"2024-03-04T16:04:51.181603+0000","last_clean":"2024-03-04T16:04:51.181603+0000","last_became_active":"2024-03-04T16:04:22.731236+0000","last_became_peered":"2024-03-04T16:04:22.731236+0000","last_unstale":"2024-03-04T16:04:51.181603+0000","last_undegraded":"2024-03-04T16:04:51.181603+0000","last_fullsized":"2024-03-04T16:04:51.181603+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:02:44.697356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172828+0000","last_change":"2024-03-04T16:04:10.580391+0000","last_active":"2024-03-04T16:04:51.172828+0000","last_peered":"2024-03-04T16:04:51.172828+0000","last_clean":"2024-03-04T16:04:51.172828+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:04:51.172828+0000","last_undegraded":"2024-03-04T16:04:51.172828+0000","last_fullsized":"2024-03-04T16:04:51.172828+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:10.580305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.580305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.834007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182196+0000","last_change":"2024-03-04T16:04:31.380258+0000","last_active":"2024-03-04T16:04:51.182196+0000","last_peered":"2024-03-04T16:04:51.182196+0000","last_clean":"2024-03-04T16:04:51.182196+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:04:51.182196+0000","last_undegraded":"2024-03-04T16:04:51.182196+0000","last_fullsized":"2024-03-04T16:04:51.182196+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:04:31.380175+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:59.306096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":89,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182543+0000","last_change":"2024-03-04T16:03:49.810121+0000","last_active":"2024-03-04T16:04:51.182543+0000","last_peered":"2024-03-04T16:04:51.182543+0000","last_clean":"2024-03-04T16:04:51.182543+0000","last_became_active":"2024-03-04T16:02:43.713331+0000","last_became_peered":"2024-03-04T16:02:43.713331+0000","last_unstale":"2024-03-04T16:04:51.182543+0000","last_undegraded":"2024-03-04T16:04:51.182543+0000","last_fullsized":"2024-03-04T16:04:51.182543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:03:49.810002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:49.810002+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.948952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180486+0000","last_change":"2024-03-04T16:03:55.306308+0000","last_active":"2024-03-04T16:04:51.180486+0000","last_peered":"2024-03-04T16:04:51.180486+0000","last_clean":"2024-03-04T16:04:51.180486+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:04:51.180486+0000","last_undegraded":"2024-03-04T16:04:51.180486+0000","last_fullsized":"2024-03-04T16:04:51.180486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:03:55.306180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.306180+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.830154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178095+0000","last_change":"2024-03-04T16:04:13.176858+0000","last_active":"2024-03-04T16:04:51.178095+0000","last_peered":"2024-03-04T16:04:51.178095+0000","last_clean":"2024-03-04T16:04:51.178095+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:04:51.178095+0000","last_undegraded":"2024-03-04T16:04:51.178095+0000","last_fullsized":"2024-03-04T16:04:51.178095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:13.176740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.176740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:28.778125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.179460+0000","last_change":"2024-03-04T16:04:14.387312+0000","last_active":"2024-03-04T16:04:51.179460+0000","last_peered":"2024-03-04T16:04:51.179460+0000","last_clean":"2024-03-04T16:04:51.179460+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:04:51.179460+0000","last_undegraded":"2024-03-04T16:04:51.179460+0000","last_fullsized":"2024-03-04T16:04:51.179460+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:14.387217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.387217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.518011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182958+0000","last_change":"2024-03-04T16:04:18.894792+0000","last_active":"2024-03-04T16:04:51.182958+0000","last_peered":"2024-03-04T16:04:51.182958+0000","last_clean":"2024-03-04T16:04:51.182958+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:04:51.182958+0000","last_undegraded":"2024-03-04T16:04:51.182958+0000","last_fullsized":"2024-03-04T16:04:51.182958+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:18.894693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:18.894693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:48.352094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183877+0000","last_change":"2024-03-04T16:04:05.755536+0000","last_active":"2024-03-04T16:04:51.183877+0000","last_peered":"2024-03-04T16:04:51.183877+0000","last_clean":"2024-03-04T16:04:51.183877+0000","last_became_active":"2024-03-04T16:02:45.745571+0000","last_became_peered":"2024-03-04T16:02:45.745571+0000","last_unstale":"2024-03-04T16:04:51.183877+0000","last_undegraded":"2024-03-04T16:04:51.183877+0000","last_fullsized":"2024-03-04T16:04:51.183877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:05.755370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:05.755370+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.469566+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183686+0000","last_change":"2024-03-04T16:04:09.871332+0000","last_active":"2024-03-04T16:04:51.183686+0000","last_peered":"2024-03-04T16:04:51.183686+0000","last_clean":"2024-03-04T16:04:51.183686+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:04:51.183686+0000","last_undegraded":"2024-03-04T16:04:51.183686+0000","last_fullsized":"2024-03-04T16:04:51.183686+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.871214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.871214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:30.539294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180757+0000","last_change":"2024-03-04T16:04:11.175190+0000","last_active":"2024-03-04T16:04:51.180757+0000","last_peered":"2024-03-04T16:04:51.180757+0000","last_clean":"2024-03-04T16:04:51.180757+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:04:51.180757+0000","last_undegraded":"2024-03-04T16:04:51.180757+0000","last_fullsized":"2024-03-04T16:04:51.180757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:11.175113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:11.175113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.876575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181429+0000","last_change":"2024-03-04T16:03:46.718298+0000","last_active":"2024-03-04T16:04:51.181429+0000","last_peered":"2024-03-04T16:04:51.181429+0000","last_clean":"2024-03-04T16:04:51.181429+0000","last_became_active":"2024-03-04T16:02:43.718241+0000","last_became_peered":"2024-03-04T16:02:43.718241+0000","last_unstale":"2024-03-04T16:04:51.181429+0000","last_undegraded":"2024-03-04T16:04:51.181429+0000","last_fullsized":"2024-03-04T16:04:51.181429+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.718188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.606455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":91,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181296+0000","last_change":"2024-03-04T16:03:52.421377+0000","last_active":"2024-03-04T16:04:51.181296+0000","last_peered":"2024-03-04T16:04:51.181296+0000","last_clean":"2024-03-04T16:04:51.181296+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:04:51.181296+0000","last_undegraded":"2024-03-04T16:04:51.181296+0000","last_fullsized":"2024-03-04T16:04:51.181296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.421285+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:19.703150+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"100'731","reported_seq":751,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.277171+0000","last_change":"2024-03-04T16:04:36.471163+0000","last_active":"2024-03-04T16:04:52.277171+0000","last_peered":"2024-03-04T16:04:52.277171+0000","last_clean":"2024-03-04T16:04:52.277171+0000","last_became_active":"2024-03-04T16:04:36.470865+0000","last_became_peered":"2024-03-04T16:04:36.470865+0000","last_unstale":"2024-03-04T16:04:52.277171+0000","last_undegraded":"2024-03-04T16:04:52.277171+0000","last_fullsized":"2024-03-04T16:04:52.277171+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:55.585054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47906816,"num_objects":731,"num_object_clones":0,"num_object_copies":2193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":731,"num_write_kb":46784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180559+0000","last_change":"2024-03-04T16:03:58.313071+0000","last_active":"2024-03-04T16:04:51.180559+0000","last_peered":"2024-03-04T16:04:51.180559+0000","last_clean":"2024-03-04T16:04:51.180559+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:04:51.180559+0000","last_undegraded":"2024-03-04T16:04:51.180559+0000","last_fullsized":"2024-03-04T16:04:51.180559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.312925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.312925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.584207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183739+0000","last_change":"2024-03-04T16:04:16.869117+0000","last_active":"2024-03-04T16:04:51.183739+0000","last_peered":"2024-03-04T16:04:51.183739+0000","last_clean":"2024-03-04T16:04:51.183739+0000","last_became_active":"2024-03-04T16:02:43.716945+0000","last_became_peered":"2024-03-04T16:02:43.716945+0000","last_unstale":"2024-03-04T16:04:51.183739+0000","last_undegraded":"2024-03-04T16:04:51.183739+0000","last_fullsized":"2024-03-04T16:04:51.183739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:04:16.868993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.868993+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.502678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180026+0000","last_change":"2024-03-04T16:03:48.260185+0000","last_active":"2024-03-04T16:04:51.180026+0000","last_peered":"2024-03-04T16:04:51.180026+0000","last_clean":"2024-03-04T16:04:51.180026+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:04:51.180026+0000","last_undegraded":"2024-03-04T16:04:51.180026+0000","last_fullsized":"2024-03-04T16:04:51.180026+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:48.260047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.260047+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:59.815204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183120+0000","last_change":"2024-03-04T16:04:27.847894+0000","last_active":"2024-03-04T16:04:51.183120+0000","last_peered":"2024-03-04T16:04:51.183120+0000","last_clean":"2024-03-04T16:04:51.183120+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:04:51.183120+0000","last_undegraded":"2024-03-04T16:04:51.183120+0000","last_fullsized":"2024-03-04T16:04:51.183120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:27.847784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:27.847784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:51.971485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"100'790","reported_seq":810,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.288685+0000","last_change":"2024-03-04T16:04:36.472659+0000","last_active":"2024-03-04T16:04:52.288685+0000","last_peered":"2024-03-04T16:04:52.288685+0000","last_clean":"2024-03-04T16:04:52.288685+0000","last_became_active":"2024-03-04T16:04:36.472312+0000","last_became_peered":"2024-03-04T16:04:36.472312+0000","last_unstale":"2024-03-04T16:04:52.288685+0000","last_undegraded":"2024-03-04T16:04:52.288685+0000","last_fullsized":"2024-03-04T16:04:52.288685+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:48.701036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":790,"num_write_kb":50560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":63,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178339+0000","last_change":"2024-03-04T16:02:49.796054+0000","last_active":"2024-03-04T16:04:51.178339+0000","last_peered":"2024-03-04T16:04:51.178339+0000","last_clean":"2024-03-04T16:04:51.178339+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:04:51.178339+0000","last_undegraded":"2024-03-04T16:04:51.178339+0000","last_fullsized":"2024-03-04T16:04:51.178339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183620+0000","last_change":"2024-03-04T16:03:55.817828+0000","last_active":"2024-03-04T16:04:51.183620+0000","last_peered":"2024-03-04T16:04:51.183620+0000","last_clean":"2024-03-04T16:04:51.183620+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:04:51.183620+0000","last_undegraded":"2024-03-04T16:04:51.183620+0000","last_fullsized":"2024-03-04T16:04:51.183620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:55.817709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:55.817709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:17.939046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":102,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172028+0000","last_change":"2024-03-04T16:03:48.407212+0000","last_active":"2024-03-04T16:04:51.172028+0000","last_peered":"2024-03-04T16:04:51.172028+0000","last_clean":"2024-03-04T16:04:51.172028+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:04:51.172028+0000","last_undegraded":"2024-03-04T16:04:51.172028+0000","last_fullsized":"2024-03-04T16:04:51.172028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:03:48.407088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:48.407088+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:54.679476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178971+0000","last_change":"2024-03-04T16:03:59.048702+0000","last_active":"2024-03-04T16:04:51.178971+0000","last_peered":"2024-03-04T16:04:51.178971+0000","last_clean":"2024-03-04T16:04:51.178971+0000","last_became_active":"2024-03-04T16:02:47.767316+0000","last_became_peered":"2024-03-04T16:02:47.767316+0000","last_unstale":"2024-03-04T16:04:51.178971+0000","last_undegraded":"2024-03-04T16:04:51.178971+0000","last_fullsized":"2024-03-04T16:04:51.178971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.048569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.048569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:04.396312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"100'744","reported_seq":762,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.287153+0000","last_change":"2024-03-04T16:04:36.472211+0000","last_active":"2024-03-04T16:04:52.287153+0000","last_peered":"2024-03-04T16:04:52.287153+0000","last_clean":"2024-03-04T16:04:52.287153+0000","last_became_active":"2024-03-04T16:04:36.471884+0000","last_became_peered":"2024-03-04T16:04:36.471884+0000","last_unstale":"2024-03-04T16:04:52.287153+0000","last_undegraded":"2024-03-04T16:04:52.287153+0000","last_fullsized":"2024-03-04T16:04:52.287153+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:06:00.184540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48758784,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":47616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182516+0000","last_change":"2024-03-04T16:04:35.817634+0000","last_active":"2024-03-04T16:04:51.182516+0000","last_peered":"2024-03-04T16:04:51.182516+0000","last_clean":"2024-03-04T16:04:51.182516+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:04:51.182516+0000","last_undegraded":"2024-03-04T16:04:51.182516+0000","last_fullsized":"2024-03-04T16:04:51.182516+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.817548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.817548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:48.688078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177140+0000","last_change":"2024-03-04T16:03:51.984808+0000","last_active":"2024-03-04T16:04:51.177140+0000","last_peered":"2024-03-04T16:04:51.177140+0000","last_clean":"2024-03-04T16:04:51.177140+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:04:51.177140+0000","last_undegraded":"2024-03-04T16:04:51.177140+0000","last_fullsized":"2024-03-04T16:04:51.177140+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:51.984721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:51.984721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:03.092310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183927+0000","last_change":"2024-03-04T16:04:24.829211+0000","last_active":"2024-03-04T16:04:51.183927+0000","last_peered":"2024-03-04T16:04:51.183927+0000","last_clean":"2024-03-04T16:04:51.183927+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:04:51.183927+0000","last_undegraded":"2024-03-04T16:04:51.183927+0000","last_fullsized":"2024-03-04T16:04:51.183927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:04:24.829107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:41.275300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181656+0000","last_change":"2024-03-04T16:04:13.665506+0000","last_active":"2024-03-04T16:04:51.181656+0000","last_peered":"2024-03-04T16:04:51.181656+0000","last_clean":"2024-03-04T16:04:51.181656+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:04:51.181656+0000","last_undegraded":"2024-03-04T16:04:51.181656+0000","last_fullsized":"2024-03-04T16:04:51.181656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:04:13.665406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.665406+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.760464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"100'761","reported_seq":781,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.275173+0000","last_change":"2024-03-04T16:04:36.467211+0000","last_active":"2024-03-04T16:04:52.275173+0000","last_peered":"2024-03-04T16:04:52.275173+0000","last_clean":"2024-03-04T16:04:52.275173+0000","last_became_active":"2024-03-04T16:04:36.466884+0000","last_became_peered":"2024-03-04T16:04:36.466884+0000","last_unstale":"2024-03-04T16:04:52.275173+0000","last_undegraded":"2024-03-04T16:04:52.275173+0000","last_fullsized":"2024-03-04T16:04:52.275173+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":761,"log_dups_size":0,"ondisk_log_size":761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:37.032591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49872896,"num_objects":761,"num_object_clones":0,"num_object_copies":2283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":48704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182622+0000","last_change":"2024-03-04T16:03:50.786884+0000","last_active":"2024-03-04T16:04:51.182622+0000","last_peered":"2024-03-04T16:04:51.182622+0000","last_clean":"2024-03-04T16:04:51.182622+0000","last_became_active":"2024-03-04T16:02:49.793023+0000","last_became_peered":"2024-03-04T16:02:49.793023+0000","last_unstale":"2024-03-04T16:04:51.182622+0000","last_undegraded":"2024-03-04T16:04:51.182622+0000","last_fullsized":"2024-03-04T16:04:51.182622+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.786805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.786805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:02.570842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176468+0000","last_change":"2024-03-04T16:04:19.170547+0000","last_active":"2024-03-04T16:04:51.176468+0000","last_peered":"2024-03-04T16:04:51.176468+0000","last_clean":"2024-03-04T16:04:51.176468+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:04:51.176468+0000","last_undegraded":"2024-03-04T16:04:51.176468+0000","last_fullsized":"2024-03-04T16:04:51.176468+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:19.170463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:19.170463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:38.808626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180680+0000","last_change":"2024-03-04T16:04:06.230119+0000","last_active":"2024-03-04T16:04:51.180680+0000","last_peered":"2024-03-04T16:04:51.180680+0000","last_clean":"2024-03-04T16:04:51.180680+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:04:51.180680+0000","last_undegraded":"2024-03-04T16:04:51.180680+0000","last_fullsized":"2024-03-04T16:04:51.180680+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:06.230030+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.230030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.804736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"75'9","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183303+0000","last_change":"2024-03-04T16:04:14.777059+0000","last_active":"2024-03-04T16:04:51.183303+0000","last_peered":"2024-03-04T16:04:51.183303+0000","last_clean":"2024-03-04T16:04:51.183303+0000","last_became_active":"2024-03-04T16:02:45.752201+0000","last_became_peered":"2024-03-04T16:02:45.752201+0000","last_unstale":"2024-03-04T16:04:51.183303+0000","last_undegraded":"2024-03-04T16:04:51.183303+0000","last_fullsized":"2024-03-04T16:04:51.183303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:04:14.776970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.776970+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.521202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"100'730","reported_seq":750,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.286350+0000","last_change":"2024-03-04T16:04:36.470799+0000","last_active":"2024-03-04T16:04:52.286350+0000","last_peered":"2024-03-04T16:04:52.286350+0000","last_clean":"2024-03-04T16:04:52.286350+0000","last_became_active":"2024-03-04T16:04:36.470388+0000","last_became_peered":"2024-03-04T16:04:36.470388+0000","last_unstale":"2024-03-04T16:04:52.286350+0000","last_undegraded":"2024-03-04T16:04:52.286350+0000","last_fullsized":"2024-03-04T16:04:52.286350+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":730,"log_dups_size":0,"ondisk_log_size":730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:06:00.621986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47841280,"num_objects":730,"num_object_clones":0,"num_object_copies":2190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":730,"num_write_kb":46720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.178362+0000","last_change":"2024-03-04T16:04:32.338377+0000","last_active":"2024-03-04T16:04:51.178362+0000","last_peered":"2024-03-04T16:04:51.178362+0000","last_clean":"2024-03-04T16:04:51.178362+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:04:51.178362+0000","last_undegraded":"2024-03-04T16:04:51.178362+0000","last_fullsized":"2024-03-04T16:04:51.178362+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:32.338288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:32.338288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:40.507201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180360+0000","last_change":"2024-03-04T16:03:54.344325+0000","last_active":"2024-03-04T16:04:51.180360+0000","last_peered":"2024-03-04T16:04:51.180360+0000","last_clean":"2024-03-04T16:04:51.180360+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:04:51.180360+0000","last_undegraded":"2024-03-04T16:04:51.180360+0000","last_fullsized":"2024-03-04T16:04:51.180360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:03:54.344193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:54.344193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.913538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172185+0000","last_change":"2024-03-04T16:03:59.437627+0000","last_active":"2024-03-04T16:04:51.172185+0000","last_peered":"2024-03-04T16:04:51.172185+0000","last_clean":"2024-03-04T16:04:51.172185+0000","last_became_active":"2024-03-04T16:02:43.717129+0000","last_became_peered":"2024-03-04T16:02:43.717129+0000","last_unstale":"2024-03-04T16:04:51.172185+0000","last_undegraded":"2024-03-04T16:04:51.172185+0000","last_fullsized":"2024-03-04T16:04:51.172185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:59.437524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:59.437524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:05.171391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":154,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185377+0000","last_change":"2024-03-04T16:04:31.091854+0000","last_active":"2024-03-04T16:04:51.185377+0000","last_peered":"2024-03-04T16:04:51.185377+0000","last_clean":"2024-03-04T16:04:51.185377+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:04:51.185377+0000","last_undegraded":"2024-03-04T16:04:51.185377+0000","last_fullsized":"2024-03-04T16:04:51.185377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:04:31.091760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:31.091760+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:58.386703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":68,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184020+0000","last_change":"2024-03-04T16:02:47.773975+0000","last_active":"2024-03-04T16:04:51.184020+0000","last_peered":"2024-03-04T16:04:51.184020+0000","last_clean":"2024-03-04T16:04:51.184020+0000","last_became_active":"2024-03-04T16:02:47.773547+0000","last_became_peered":"2024-03-04T16:02:47.773547+0000","last_unstale":"2024-03-04T16:04:51.184020+0000","last_undegraded":"2024-03-04T16:04:51.184020+0000","last_fullsized":"2024-03-04T16:04:51.184020+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:02:46.732387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:53.612236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"100'773","reported_seq":793,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.279161+0000","last_change":"2024-03-04T16:04:36.472028+0000","last_active":"2024-03-04T16:04:52.279161+0000","last_peered":"2024-03-04T16:04:52.279161+0000","last_clean":"2024-03-04T16:04:52.279161+0000","last_became_active":"2024-03-04T16:04:36.471714+0000","last_became_peered":"2024-03-04T16:04:36.471714+0000","last_unstale":"2024-03-04T16:04:52.279161+0000","last_undegraded":"2024-03-04T16:04:52.279161+0000","last_fullsized":"2024-03-04T16:04:52.279161+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:45.178851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50659328,"num_objects":773,"num_object_clones":0,"num_object_copies":2319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":773,"num_write_kb":49472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183994+0000","last_change":"2024-03-04T16:04:15.888858+0000","last_active":"2024-03-04T16:04:51.183994+0000","last_peered":"2024-03-04T16:04:51.183994+0000","last_clean":"2024-03-04T16:04:51.183994+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:04:51.183994+0000","last_undegraded":"2024-03-04T16:04:51.183994+0000","last_fullsized":"2024-03-04T16:04:51.183994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.888765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.888765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.962116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183745+0000","last_change":"2024-03-04T16:04:15.078996+0000","last_active":"2024-03-04T16:04:51.183745+0000","last_peered":"2024-03-04T16:04:51.183745+0000","last_clean":"2024-03-04T16:04:51.183745+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:04:51.183745+0000","last_undegraded":"2024-03-04T16:04:51.183745+0000","last_fullsized":"2024-03-04T16:04:51.183745+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.078890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.078890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.844807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":93,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176559+0000","last_change":"2024-03-04T16:03:47.979184+0000","last_active":"2024-03-04T16:04:51.176559+0000","last_peered":"2024-03-04T16:04:51.176559+0000","last_clean":"2024-03-04T16:04:51.176559+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:04:51.176559+0000","last_undegraded":"2024-03-04T16:04:51.176559+0000","last_fullsized":"2024-03-04T16:04:51.176559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:47.979058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:03:47.979058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.315882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":111,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184890+0000","last_change":"2024-03-04T16:04:17.030150+0000","last_active":"2024-03-04T16:04:51.184890+0000","last_peered":"2024-03-04T16:04:51.184890+0000","last_clean":"2024-03-04T16:04:51.184890+0000","last_became_active":"2024-03-04T16:02:45.748615+0000","last_became_peered":"2024-03-04T16:02:45.748615+0000","last_unstale":"2024-03-04T16:04:51.184890+0000","last_undegraded":"2024-03-04T16:04:51.184890+0000","last_fullsized":"2024-03-04T16:04:51.184890+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:04:17.030053+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:45.396813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177656+0000","last_change":"2024-03-04T16:04:03.048703+0000","last_active":"2024-03-04T16:04:51.177656+0000","last_peered":"2024-03-04T16:04:51.177656+0000","last_clean":"2024-03-04T16:04:51.177656+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:04:51.177656+0000","last_undegraded":"2024-03-04T16:04:51.177656+0000","last_fullsized":"2024-03-04T16:04:51.177656+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:03.048620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:03.048620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.405283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"100'754","reported_seq":774,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.285180+0000","last_change":"2024-03-04T16:04:36.469944+0000","last_active":"2024-03-04T16:04:52.285180+0000","last_peered":"2024-03-04T16:04:52.285180+0000","last_clean":"2024-03-04T16:04:52.285180+0000","last_became_active":"2024-03-04T16:04:36.469646+0000","last_became_peered":"2024-03-04T16:04:36.469646+0000","last_unstale":"2024-03-04T16:04:52.285180+0000","last_undegraded":"2024-03-04T16:04:52.285180+0000","last_fullsized":"2024-03-04T16:04:52.285180+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":754,"log_dups_size":0,"ondisk_log_size":754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:49.520320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49414144,"num_objects":754,"num_object_clones":0,"num_object_copies":2262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":48256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180736+0000","last_change":"2024-03-04T16:04:16.164295+0000","last_active":"2024-03-04T16:04:51.180736+0000","last_peered":"2024-03-04T16:04:51.180736+0000","last_clean":"2024-03-04T16:04:51.180736+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:04:51.180736+0000","last_undegraded":"2024-03-04T16:04:51.180736+0000","last_fullsized":"2024-03-04T16:04:51.180736+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.164189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.164189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:27.368351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":336,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182653+0000","last_change":"2024-03-04T16:03:12.851794+0000","last_active":"2024-03-04T16:04:51.182653+0000","last_peered":"2024-03-04T16:04:51.182653+0000","last_clean":"2024-03-04T16:04:51.182653+0000","last_became_active":"2024-03-04T16:02:34.079164+0000","last_became_peered":"2024-03-04T16:02:34.079164+0000","last_unstale":"2024-03-04T16:04:51.182653+0000","last_undegraded":"2024-03-04T16:04:51.182653+0000","last_fullsized":"2024-03-04T16:04:51.182653+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'119","last_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:03:12.851677+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:04:55.920427+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177485+0000","last_change":"2024-03-04T16:03:57.445763+0000","last_active":"2024-03-04T16:04:51.177485+0000","last_peered":"2024-03-04T16:04:51.177485+0000","last_clean":"2024-03-04T16:04:51.177485+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:04:51.177485+0000","last_undegraded":"2024-03-04T16:04:51.177485+0000","last_fullsized":"2024-03-04T16:04:51.177485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.445659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.445659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.402395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":128,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.182454+0000","last_change":"2024-03-04T16:03:58.764928+0000","last_active":"2024-03-04T16:04:51.182454+0000","last_peered":"2024-03-04T16:04:51.182454+0000","last_clean":"2024-03-04T16:04:51.182454+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:04:51.182454+0000","last_undegraded":"2024-03-04T16:04:51.182454+0000","last_fullsized":"2024-03-04T16:04:51.182454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:03:58.764821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.764821+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.352281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183532+0000","last_change":"2024-03-04T16:03:52.917188+0000","last_active":"2024-03-04T16:04:51.183532+0000","last_peered":"2024-03-04T16:04:51.183532+0000","last_clean":"2024-03-04T16:04:51.183532+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:04:51.183532+0000","last_undegraded":"2024-03-04T16:04:51.183532+0000","last_fullsized":"2024-03-04T16:04:51.183532+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.917090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.917090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:00.018319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"100'745","reported_seq":765,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.279226+0000","last_change":"2024-03-04T16:04:36.465705+0000","last_active":"2024-03-04T16:04:52.279226+0000","last_peered":"2024-03-04T16:04:52.279226+0000","last_clean":"2024-03-04T16:04:52.279226+0000","last_became_active":"2024-03-04T16:04:36.465393+0000","last_became_peered":"2024-03-04T16:04:36.465393+0000","last_unstale":"2024-03-04T16:04:52.279226+0000","last_undegraded":"2024-03-04T16:04:52.279226+0000","last_fullsized":"2024-03-04T16:04:52.279226+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:06:02.993947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184134+0000","last_change":"2024-03-04T16:03:52.944487+0000","last_active":"2024-03-04T16:04:51.184134+0000","last_peered":"2024-03-04T16:04:51.184134+0000","last_clean":"2024-03-04T16:04:51.184134+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:04:51.184134+0000","last_undegraded":"2024-03-04T16:04:51.184134+0000","last_fullsized":"2024-03-04T16:04:51.184134+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:03:52.944410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.132708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183037+0000","last_change":"2024-03-04T16:04:09.738590+0000","last_active":"2024-03-04T16:04:51.183037+0000","last_peered":"2024-03-04T16:04:51.183037+0000","last_clean":"2024-03-04T16:04:51.183037+0000","last_became_active":"2024-03-04T16:02:43.713276+0000","last_became_peered":"2024-03-04T16:02:43.713276+0000","last_unstale":"2024-03-04T16:04:51.183037+0000","last_undegraded":"2024-03-04T16:04:51.183037+0000","last_fullsized":"2024-03-04T16:04:51.183037+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:09.738490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.738490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.085127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":219,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.193662+0000","last_change":"2024-03-04T16:04:07.872620+0000","last_active":"2024-03-04T16:04:51.193662+0000","last_peered":"2024-03-04T16:04:51.193662+0000","last_clean":"2024-03-04T16:04:51.193662+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:04:51.193662+0000","last_undegraded":"2024-03-04T16:04:51.193662+0000","last_fullsized":"2024-03-04T16:04:51.193662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-04T16:04:07.872489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.872489+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:14.219006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":123,"num_read_kb":118,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":97,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183214+0000","last_change":"2024-03-04T16:04:22.685946+0000","last_active":"2024-03-04T16:04:51.183214+0000","last_peered":"2024-03-04T16:04:51.183214+0000","last_clean":"2024-03-04T16:04:51.183214+0000","last_became_active":"2024-03-04T16:02:45.755435+0000","last_became_peered":"2024-03-04T16:02:45.755435+0000","last_unstale":"2024-03-04T16:04:51.183214+0000","last_undegraded":"2024-03-04T16:04:51.183214+0000","last_fullsized":"2024-03-04T16:04:51.183214+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:04:22.685789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:22.685789+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.840772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181006+0000","last_change":"2024-03-04T16:03:57.296517+0000","last_active":"2024-03-04T16:04:51.181006+0000","last_peered":"2024-03-04T16:04:51.181006+0000","last_clean":"2024-03-04T16:04:51.181006+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:04:51.181006+0000","last_undegraded":"2024-03-04T16:04:51.181006+0000","last_fullsized":"2024-03-04T16:04:51.181006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:57.296421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:57.296421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:24.272283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"100'758","reported_seq":777,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.261100+0000","last_change":"2024-03-04T16:04:36.471137+0000","last_active":"2024-03-04T16:04:52.261100+0000","last_peered":"2024-03-04T16:04:52.261100+0000","last_clean":"2024-03-04T16:04:52.261100+0000","last_became_active":"2024-03-04T16:04:36.470468+0000","last_became_peered":"2024-03-04T16:04:36.470468+0000","last_unstale":"2024-03-04T16:04:52.261100+0000","last_undegraded":"2024-03-04T16:04:52.261100+0000","last_fullsized":"2024-03-04T16:04:52.261100+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:06:02.081802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183589+0000","last_change":"2024-03-04T16:04:33.988167+0000","last_active":"2024-03-04T16:04:51.183589+0000","last_peered":"2024-03-04T16:04:51.183589+0000","last_clean":"2024-03-04T16:04:51.183589+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:04:51.183589+0000","last_undegraded":"2024-03-04T16:04:51.183589+0000","last_fullsized":"2024-03-04T16:04:51.183589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:33.988088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:33.988088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:34.145119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185281+0000","last_change":"2024-03-04T16:04:32.066420+0000","last_active":"2024-03-04T16:04:51.185281+0000","last_peered":"2024-03-04T16:04:51.185281+0000","last_clean":"2024-03-04T16:04:51.185281+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:04:51.185281+0000","last_undegraded":"2024-03-04T16:04:51.185281+0000","last_fullsized":"2024-03-04T16:04:51.185281+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:32.066332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:32.066332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:54.186865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"75'19","reported_seq":116,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183192+0000","last_change":"2024-03-04T16:04:10.746331+0000","last_active":"2024-03-04T16:04:51.183192+0000","last_peered":"2024-03-04T16:04:51.183192+0000","last_clean":"2024-03-04T16:04:51.183192+0000","last_became_active":"2024-03-04T16:02:45.744172+0000","last_became_peered":"2024-03-04T16:02:45.744172+0000","last_unstale":"2024-03-04T16:04:51.183192+0000","last_undegraded":"2024-03-04T16:04:51.183192+0000","last_fullsized":"2024-03-04T16:04:51.183192+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:04:10.746233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:10.746233+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:36.591472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181658+0000","last_change":"2024-03-04T16:03:50.770141+0000","last_active":"2024-03-04T16:04:51.181658+0000","last_peered":"2024-03-04T16:04:51.181658+0000","last_clean":"2024-03-04T16:04:51.181658+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:04:51.181658+0000","last_undegraded":"2024-03-04T16:04:51.181658+0000","last_fullsized":"2024-03-04T16:04:51.181658+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:50.770048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:03:50.770048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:16.417482+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"100'788","reported_seq":808,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.293582+0000","last_change":"2024-03-04T16:04:36.467788+0000","last_active":"2024-03-04T16:04:52.293582+0000","last_peered":"2024-03-04T16:04:52.293582+0000","last_clean":"2024-03-04T16:04:52.293582+0000","last_became_active":"2024-03-04T16:04:36.467504+0000","last_became_peered":"2024-03-04T16:04:36.467504+0000","last_unstale":"2024-03-04T16:04:52.293582+0000","last_undegraded":"2024-03-04T16:04:52.293582+0000","last_fullsized":"2024-03-04T16:04:52.293582+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":788,"log_dups_size":0,"ondisk_log_size":788,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.525880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":72,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181036+0000","last_change":"2024-03-04T16:04:32.662671+0000","last_active":"2024-03-04T16:04:51.181036+0000","last_peered":"2024-03-04T16:04:51.181036+0000","last_clean":"2024-03-04T16:04:51.181036+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:04:51.181036+0000","last_undegraded":"2024-03-04T16:04:51.181036+0000","last_fullsized":"2024-03-04T16:04:51.181036+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:32.662558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:32.662558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:39.197949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180435+0000","last_change":"2024-03-04T16:04:15.206400+0000","last_active":"2024-03-04T16:04:51.180435+0000","last_peered":"2024-03-04T16:04:51.180435+0000","last_clean":"2024-03-04T16:04:51.180435+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:04:51.180435+0000","last_undegraded":"2024-03-04T16:04:51.180435+0000","last_fullsized":"2024-03-04T16:04:51.180435+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:15.206292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:15.206292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:22.696036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"75'10","reported_seq":95,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185745+0000","last_change":"2024-03-04T16:04:14.060813+0000","last_active":"2024-03-04T16:04:51.185745+0000","last_peered":"2024-03-04T16:04:51.185745+0000","last_clean":"2024-03-04T16:04:51.185745+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:04:51.185745+0000","last_undegraded":"2024-03-04T16:04:51.185745+0000","last_fullsized":"2024-03-04T16:04:51.185745+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:04:14.060715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:14.060715+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:30.595410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"75'8","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.184031+0000","last_change":"2024-03-04T16:04:29.632233+0000","last_active":"2024-03-04T16:04:51.184031+0000","last_peered":"2024-03-04T16:04:51.184031+0000","last_clean":"2024-03-04T16:04:51.184031+0000","last_became_active":"2024-03-04T16:02:47.777462+0000","last_became_peered":"2024-03-04T16:02:47.777462+0000","last_unstale":"2024-03-04T16:04:51.184031+0000","last_undegraded":"2024-03-04T16:04:51.184031+0000","last_fullsized":"2024-03-04T16:04:51.184031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:29.632142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:29.632142+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.939299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"100'783","reported_seq":803,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.280544+0000","last_change":"2024-03-04T16:04:36.473547+0000","last_active":"2024-03-04T16:04:52.280544+0000","last_peered":"2024-03-04T16:04:52.280544+0000","last_clean":"2024-03-04T16:04:52.280544+0000","last_became_active":"2024-03-04T16:04:36.473063+0000","last_became_peered":"2024-03-04T16:04:36.473063+0000","last_unstale":"2024-03-04T16:04:52.280544+0000","last_undegraded":"2024-03-04T16:04:52.280544+0000","last_fullsized":"2024-03-04T16:04:52.280544+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":783,"log_dups_size":0,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:59.706988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51314688,"num_objects":783,"num_object_clones":0,"num_object_copies":2349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":783,"num_write_kb":50112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183347+0000","last_change":"2024-03-04T16:03:56.858487+0000","last_active":"2024-03-04T16:04:51.183347+0000","last_peered":"2024-03-04T16:04:51.183347+0000","last_clean":"2024-03-04T16:04:51.183347+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:04:51.183347+0000","last_undegraded":"2024-03-04T16:04:51.183347+0000","last_fullsized":"2024-03-04T16:04:51.183347+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:56.858376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:03:56.858376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:11.769864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.183317+0000","last_change":"2024-03-04T16:04:16.701272+0000","last_active":"2024-03-04T16:04:51.183317+0000","last_peered":"2024-03-04T16:04:51.183317+0000","last_clean":"2024-03-04T16:04:51.183317+0000","last_became_active":"2024-03-04T16:02:43.712287+0000","last_became_peered":"2024-03-04T16:02:43.712287+0000","last_unstale":"2024-03-04T16:04:51.183317+0000","last_undegraded":"2024-03-04T16:04:51.183317+0000","last_fullsized":"2024-03-04T16:04:51.183317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:04:16.701175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.701175+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.474917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":108,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171975+0000","last_change":"2024-03-04T16:04:06.515940+0000","last_active":"2024-03-04T16:04:51.171975+0000","last_peered":"2024-03-04T16:04:51.171975+0000","last_clean":"2024-03-04T16:04:51.171975+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:04:51.171975+0000","last_undegraded":"2024-03-04T16:04:51.171975+0000","last_fullsized":"2024-03-04T16:04:51.171975+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:04:06.515823+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:33.602986+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":110,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177719+0000","last_change":"2024-03-04T16:04:09.344572+0000","last_active":"2024-03-04T16:04:51.177719+0000","last_peered":"2024-03-04T16:04:51.177719+0000","last_clean":"2024-03-04T16:04:51.177719+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:04:51.177719+0000","last_undegraded":"2024-03-04T16:04:51.177719+0000","last_fullsized":"2024-03-04T16:04:51.177719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:09.344448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.344448+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:23.358656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"100'801","reported_seq":821,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.283152+0000","last_change":"2024-03-04T16:04:36.474294+0000","last_active":"2024-03-04T16:04:52.283152+0000","last_peered":"2024-03-04T16:04:52.283152+0000","last_clean":"2024-03-04T16:04:52.283152+0000","last_became_active":"2024-03-04T16:04:36.473541+0000","last_became_peered":"2024-03-04T16:04:36.473541+0000","last_unstale":"2024-03-04T16:04:52.283152+0000","last_undegraded":"2024-03-04T16:04:52.283152+0000","last_fullsized":"2024-03-04T16:04:52.283152+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:51.289388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":51264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171788+0000","last_change":"2024-03-04T16:04:30.701960+0000","last_active":"2024-03-04T16:04:51.171788+0000","last_peered":"2024-03-04T16:04:51.171788+0000","last_clean":"2024-03-04T16:04:51.171788+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:04:51.171788+0000","last_undegraded":"2024-03-04T16:04:51.171788+0000","last_fullsized":"2024-03-04T16:04:51.171788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:30.701884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:30.701884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:35.944086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.179257+0000","last_change":"2024-03-04T16:04:18.352181+0000","last_active":"2024-03-04T16:04:51.179257+0000","last_peered":"2024-03-04T16:04:51.179257+0000","last_clean":"2024-03-04T16:04:51.179257+0000","last_became_active":"2024-03-04T16:02:43.715346+0000","last_became_peered":"2024-03-04T16:02:43.715346+0000","last_unstale":"2024-03-04T16:04:51.179257+0000","last_undegraded":"2024-03-04T16:04:51.179257+0000","last_fullsized":"2024-03-04T16:04:51.179257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:18.352089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:04:18.352089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:40.527904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":112,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177408+0000","last_change":"2024-03-04T16:04:23.403113+0000","last_active":"2024-03-04T16:04:51.177408+0000","last_peered":"2024-03-04T16:04:51.177408+0000","last_clean":"2024-03-04T16:04:51.177408+0000","last_became_active":"2024-03-04T16:02:45.749633+0000","last_became_peered":"2024-03-04T16:02:45.749633+0000","last_unstale":"2024-03-04T16:04:51.177408+0000","last_undegraded":"2024-03-04T16:04:51.177408+0000","last_fullsized":"2024-03-04T16:04:51.177408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:04:23.403015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:23.403015+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.013014+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":76,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171799+0000","last_change":"2024-03-04T16:04:19.677401+0000","last_active":"2024-03-04T16:04:51.171799+0000","last_peered":"2024-03-04T16:04:51.171799+0000","last_clean":"2024-03-04T16:04:51.171799+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:04:51.171799+0000","last_undegraded":"2024-03-04T16:04:51.171799+0000","last_fullsized":"2024-03-04T16:04:51.171799+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:19.677319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:19.677319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:21.711702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"100'803","reported_seq":821,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.293214+0000","last_change":"2024-03-04T16:04:36.470816+0000","last_active":"2024-03-04T16:04:52.293214+0000","last_peered":"2024-03-04T16:04:52.293214+0000","last_clean":"2024-03-04T16:04:52.293214+0000","last_became_active":"2024-03-04T16:04:36.470460+0000","last_became_peered":"2024-03-04T16:04:36.470460+0000","last_unstale":"2024-03-04T16:04:52.293214+0000","last_undegraded":"2024-03-04T16:04:52.293214+0000","last_fullsized":"2024-03-04T16:04:52.293214+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:06:01.472265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52625408,"num_objects":803,"num_object_clones":0,"num_object_copies":2409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":803,"num_write_kb":51392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.180260+0000","last_change":"2024-03-04T16:04:18.117442+0000","last_active":"2024-03-04T16:04:51.180260+0000","last_peered":"2024-03-04T16:04:51.180260+0000","last_clean":"2024-03-04T16:04:51.180260+0000","last_became_active":"2024-03-04T16:02:49.791851+0000","last_became_peered":"2024-03-04T16:02:49.791851+0000","last_unstale":"2024-03-04T16:04:51.180260+0000","last_undegraded":"2024-03-04T16:04:51.180260+0000","last_fullsized":"2024-03-04T16:04:51.180260+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:18.117331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:18.117331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:36.845466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":109,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.173052+0000","last_change":"2024-03-04T16:04:21.714877+0000","last_active":"2024-03-04T16:04:51.173052+0000","last_peered":"2024-03-04T16:04:51.173052+0000","last_clean":"2024-03-04T16:04:51.173052+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:04:51.173052+0000","last_undegraded":"2024-03-04T16:04:51.173052+0000","last_fullsized":"2024-03-04T16:04:51.173052+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:04:21.714745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:21.714745+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:29.136186+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177309+0000","last_change":"2024-03-04T16:03:58.462106+0000","last_active":"2024-03-04T16:04:51.177309+0000","last_peered":"2024-03-04T16:04:51.177309+0000","last_clean":"2024-03-04T16:04:51.177309+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:04:51.177309+0000","last_undegraded":"2024-03-04T16:04:51.177309+0000","last_fullsized":"2024-03-04T16:04:51.177309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:03:58.462018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:03:58.462018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:08.477439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":104,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185063+0000","last_change":"2024-03-04T16:03:46.963168+0000","last_active":"2024-03-04T16:04:51.185063+0000","last_peered":"2024-03-04T16:04:51.185063+0000","last_clean":"2024-03-04T16:04:51.185063+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:04:51.185063+0000","last_undegraded":"2024-03-04T16:04:51.185063+0000","last_fullsized":"2024-03-04T16:04:51.185063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:03:46.963065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:03:46.963065+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:09.801631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"100'765","reported_seq":785,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.290984+0000","last_change":"2024-03-04T16:04:36.468127+0000","last_active":"2024-03-04T16:04:52.290984+0000","last_peered":"2024-03-04T16:04:52.290984+0000","last_clean":"2024-03-04T16:04:52.290984+0000","last_became_active":"2024-03-04T16:04:36.467867+0000","last_became_peered":"2024-03-04T16:04:36.467867+0000","last_unstale":"2024-03-04T16:04:52.290984+0000","last_undegraded":"2024-03-04T16:04:52.290984+0000","last_fullsized":"2024-03-04T16:04:52.290984+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":765,"log_dups_size":0,"ondisk_log_size":765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:45.162510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":765,"num_write_kb":48960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":64,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181496+0000","last_change":"2024-03-04T16:02:49.794003+0000","last_active":"2024-03-04T16:04:51.181496+0000","last_peered":"2024-03-04T16:04:51.181496+0000","last_clean":"2024-03-04T16:04:51.181496+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:04:51.181496+0000","last_undegraded":"2024-03-04T16:04:51.181496+0000","last_fullsized":"2024-03-04T16:04:51.181496+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:02:48.753581+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181000+0000","last_change":"2024-03-04T16:04:13.194800+0000","last_active":"2024-03-04T16:04:51.181000+0000","last_peered":"2024-03-04T16:04:51.181000+0000","last_clean":"2024-03-04T16:04:51.181000+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:04:51.181000+0000","last_undegraded":"2024-03-04T16:04:51.181000+0000","last_fullsized":"2024-03-04T16:04:51.181000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:13.194723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:13.194723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:20.282552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.171753+0000","last_change":"2024-03-04T16:04:29.654109+0000","last_active":"2024-03-04T16:04:51.171753+0000","last_peered":"2024-03-04T16:04:51.171753+0000","last_clean":"2024-03-04T16:04:51.171753+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:04:51.171753+0000","last_undegraded":"2024-03-04T16:04:51.171753+0000","last_fullsized":"2024-03-04T16:04:51.171753+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:04:29.654032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.879373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"75'11","reported_seq":98,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172958+0000","last_change":"2024-03-04T16:04:22.690985+0000","last_active":"2024-03-04T16:04:51.172958+0000","last_peered":"2024-03-04T16:04:51.172958+0000","last_clean":"2024-03-04T16:04:51.172958+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:04:51.172958+0000","last_undegraded":"2024-03-04T16:04:51.172958+0000","last_fullsized":"2024-03-04T16:04:51.172958+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:04:22.690898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:22.690898+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:34.757701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"100'791","reported_seq":811,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.285389+0000","last_change":"2024-03-04T16:04:36.468101+0000","last_active":"2024-03-04T16:04:52.285389+0000","last_peered":"2024-03-04T16:04:52.285389+0000","last_clean":"2024-03-04T16:04:52.285389+0000","last_became_active":"2024-03-04T16:04:36.467742+0000","last_became_peered":"2024-03-04T16:04:36.467742+0000","last_unstale":"2024-03-04T16:04:52.285389+0000","last_undegraded":"2024-03-04T16:04:52.285389+0000","last_fullsized":"2024-03-04T16:04:52.285389+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:39.137129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51838976,"num_objects":791,"num_object_clones":0,"num_object_copies":2373,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":791,"num_write_kb":50624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177479+0000","last_change":"2024-03-04T16:04:07.063161+0000","last_active":"2024-03-04T16:04:51.177479+0000","last_peered":"2024-03-04T16:04:51.177479+0000","last_clean":"2024-03-04T16:04:51.177479+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:04:51.177479+0000","last_undegraded":"2024-03-04T16:04:51.177479+0000","last_fullsized":"2024-03-04T16:04:51.177479+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.063063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.063063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:07.152553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":75,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.185431+0000","last_change":"2024-03-04T16:04:04.976057+0000","last_active":"2024-03-04T16:04:51.185431+0000","last_peered":"2024-03-04T16:04:51.185431+0000","last_clean":"2024-03-04T16:04:51.185431+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:04:51.185431+0000","last_undegraded":"2024-03-04T16:04:51.185431+0000","last_fullsized":"2024-03-04T16:04:51.185431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.975944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.975944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.116230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.181764+0000","last_change":"2024-03-04T16:04:07.763042+0000","last_active":"2024-03-04T16:04:51.181764+0000","last_peered":"2024-03-04T16:04:51.181764+0000","last_clean":"2024-03-04T16:04:51.181764+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:04:51.181764+0000","last_undegraded":"2024-03-04T16:04:51.181764+0000","last_fullsized":"2024-03-04T16:04:51.181764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:04:07.762953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:37.004167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":107,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172669+0000","last_change":"2024-03-04T16:04:09.533365+0000","last_active":"2024-03-04T16:04:51.172669+0000","last_peered":"2024-03-04T16:04:51.172669+0000","last_clean":"2024-03-04T16:04:51.172669+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:04:51.172669+0000","last_undegraded":"2024-03-04T16:04:51.172669+0000","last_fullsized":"2024-03-04T16:04:51.172669+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:04:09.533267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:09.533267+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:12.228405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"100'800","reported_seq":820,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:52.285413+0000","last_change":"2024-03-04T16:04:36.470115+0000","last_active":"2024-03-04T16:04:52.285413+0000","last_peered":"2024-03-04T16:04:52.285413+0000","last_clean":"2024-03-04T16:04:52.285413+0000","last_became_active":"2024-03-04T16:04:36.469818+0000","last_became_peered":"2024-03-04T16:04:36.469818+0000","last_unstale":"2024-03-04T16:04:52.285413+0000","last_undegraded":"2024-03-04T16:04:52.285413+0000","last_fullsized":"2024-03-04T16:04:52.285413+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:35.431433+0000","last_clean_scrub_stamp":"2024-03-04T16:04:35.431433+0000","objects_scrubbed":0,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:53.396377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":51200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":71,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.173107+0000","last_change":"2024-03-04T16:04:04.509748+0000","last_active":"2024-03-04T16:04:51.173107+0000","last_peered":"2024-03-04T16:04:51.173107+0000","last_clean":"2024-03-04T16:04:51.173107+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:04:51.173107+0000","last_undegraded":"2024-03-04T16:04:51.173107+0000","last_fullsized":"2024-03-04T16:04:51.173107+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:04.509608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:04:04.509608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:26.020538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":113,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.177784+0000","last_change":"2024-03-04T16:04:18.186936+0000","last_active":"2024-03-04T16:04:51.177784+0000","last_peered":"2024-03-04T16:04:51.177784+0000","last_clean":"2024-03-04T16:04:51.177784+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:04:51.177784+0000","last_undegraded":"2024-03-04T16:04:51.177784+0000","last_fullsized":"2024-03-04T16:04:51.177784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-04T16:04:18.186834+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:04:18.186834+0000","last_clean_scrub_stamp":"2024-03-04T16:04:18.186834+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:32.711572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":83,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.176636+0000","last_change":"2024-03-04T16:04:16.373386+0000","last_active":"2024-03-04T16:04:51.176636+0000","last_peered":"2024-03-04T16:04:51.176636+0000","last_clean":"2024-03-04T16:04:51.176636+0000","last_became_active":"2024-03-04T16:02:43.714792+0000","last_became_peered":"2024-03-04T16:02:43.714792+0000","last_unstale":"2024-03-04T16:04:51.176636+0000","last_undegraded":"2024-03-04T16:04:51.176636+0000","last_fullsized":"2024-03-04T16:04:51.176636+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:04:16.373275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:43.726905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":94,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-04T16:04:51.172824+0000","last_change":"2024-03-04T16:04:20.698459+0000","last_active":"2024-03-04T16:04:51.172824+0000","last_peered":"2024-03-04T16:04:51.172824+0000","last_clean":"2024-03-04T16:04:51.172824+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:04:51.172824+0000","last_undegraded":"2024-03-04T16:04:51.172824+0000","last_fullsized":"2024-03-04T16:04:51.172824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:04:20.698364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:04:20.698364+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:05:41.056478+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":807206912,"num_objects":12317,"num_object_clones":0,"num_object_copies":36951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12317,"num_write_kb":788288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2422210560,"data_stored":2422210560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12317,"ondisk_log_size":12317,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":416,"ondisk_log_size":416,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070494,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253520,"kb_used_data":226500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93491632,"statfs":{"total":95995035648,"available":95735431168,"internally_reserved":0,"allocated":231936000,"data_stored":231550223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364834,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245304,"kb_used_data":218284,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93499848,"statfs":{"total":95995035648,"available":95743844352,"internally_reserved":0,"allocated":223522816,"data_stored":223159303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312552,"kb_used_data":268892,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93432600,"statfs":{"total":95995035648,"available":95674982400,"internally_reserved":0,"allocated":275345408,"data_stored":274933751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738432,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360528,"kb_used_data":333508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93384624,"statfs":{"total":95995035648,"available":95625854976,"internally_reserved":0,"allocated":341512192,"data_stored":341125759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320440,"kb_used_data":276780,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93424712,"statfs":{"total":95995035648,"available":95666905088,"internally_reserved":0,"allocated":283422720,"data_stored":283059383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149745,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":470744,"kb_used_data":439568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93274408,"statfs":{"total":95995035648,"available":95512993792,"internally_reserved":0,"allocated":450117632,"data_stored":449718477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256400,"kb_used_data":216900,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93488752,"statfs":{"total":95995035648,"available":95732482048,"internally_reserved":0,"allocated":222105600,"data_stored":221717957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626472,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":197432,"kb_used_data":157932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93547720,"statfs":{"total":95995035648,"available":95792865280,"internally_reserved":0,"allocated":161722368,"data_stored":161358637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":399900672,"data_stored":399900672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298450944,"data_stored":298450944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":306118656,"data_stored":306118656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":459341824,"data_stored":459341824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":253558784,"data_stored":253558784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":201326592,"data_stored":201326592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":250347520,"data_stored":250347520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253165568,"data_stored":253165568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:04:59.572 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-04T16:04:59.614 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:04:59.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:04:59.687 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:04:59.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:04:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:04:59.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:04:59.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 12 16 28587 28571 148.788 125.812 0.00309916 0.00671572 2024-03-04T16:04:59.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:04:59.968 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:04:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:04:59.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:00.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:00.117 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:00.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:00.250 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:00.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.385 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:00.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:00.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.530 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:00.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:00.610 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:00 smithi038 ceph-mon[29779]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 840 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-04T16:05:00.610 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:00 smithi038 ceph-mon[38113]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 840 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-04T16:05:00.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:00.675 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:00.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:00 smithi176 ceph-mon[33619]: pgmap v75: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 840 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-04T16:05:00.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:00.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 13 16 30621 30605 147.121 127.125 0.00358427 0.00679165 2024-03-04T16:05:00.942 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:00.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:01.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:01.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:01.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:01.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:01.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.375 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:01.377 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:01.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:01.508 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:05:01.508 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:05:01.509 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:05:01.509 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:05:01.520 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:01.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.646 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:01.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:01.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:05:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:05:01.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23118 7.6 7:698e4874:::benchmark_data_smithi038_77863_object23117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.951477+0000", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7302817470000003, 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.085754825000000007, 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23118 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951477+0000", 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951477+0000", 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951478+0000", 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7999999999999995e-07 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951498+0000", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0332000000000001e-05 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951504+0000", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5989999999999998e-06 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.951511+0000", 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.674e-06 2024-03-04T16:05:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.953072+0000", 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00156051 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.953253+0000", 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018101799999999999 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.953385+0000", 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013280099999999999 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955214+0000", 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001828297 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955225+0000", 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1549000000000001e-05 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037113+0000", 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.081887615999999996 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037163+0000", 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9901999999999998e-05 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037232+0000", 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8847e-05 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23124 7.f 7:f923f0ef:::benchmark_data_smithi038_77863_object23123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.954490+0000", 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.727268563, 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.081133285999999999, 2024-03-04T16:05:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23124 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954490+0000", 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954490+0000", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954492+0000", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8109999999999999e-06 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954514+0000", 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2603999999999999e-05 2024-03-04T16:05:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954524+0000", 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7529999999999996e-06 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.954533+0000", 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0159999999999997e-06 2024-03-04T16:05:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956209+0000", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00167588 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956884+0000", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00067471299999999996 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.957034+0000", 2024-03-04T16:05:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014970599999999999 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958453+0000", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001419729 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959003+0000", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00054975900000000003 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035507+0000", 2024-03-04T16:05:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.076503631000000002 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035569+0000", 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2549e-05 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035623+0000", 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4135000000000003e-05 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23125 7.6 7:63021a9f:::benchmark_data_smithi038_77863_object23124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.955168+0000", 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7265908469999998, 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.082306636000000002, 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23125 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955168+0000", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955168+0000", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955169+0000", 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.263e-06 2024-03-04T16:05:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955195+0000", 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6364e-05 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955206+0000", 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1025e-05 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955214+0000", 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8990000000000001e-06 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955247+0000", 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3185000000000002e-05 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955351+0000", 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010362299999999999 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.955476+0000", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000124623 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.957460+0000", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00198401 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.957546+0000", 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6704000000000002e-05 2024-03-04T16:05:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037380+0000", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079833124000000005 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037432+0000", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.257e-05 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037474+0000", 2024-03-04T16:05:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2246000000000002e-05 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23129 7.6 7:6b3b5098:::benchmark_data_smithi038_77863_object23128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.956580+0000", 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.725178691, 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.082955101000000003, 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23129 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956580+0000", 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956580+0000", 2024-03-04T16:05:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956581+0000", 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.198e-06 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956601+0000", 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9428000000000001e-05 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956609+0000", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0139999999999998e-06 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956615+0000", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2620000000000004e-06 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.957643+0000", 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010279169999999999 2024-03-04T16:05:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.957773+0000", 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012979199999999999 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958187+0000", 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041411699999999998 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037076+0000", 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.078889363000000004 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037278+0000", 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000202308 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039431+0000", 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021530120000000002 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039496+0000", 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4498000000000005e-05 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039535+0000", 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9192e-05 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.948 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23130 7.6 7:65251d05:::benchmark_data_smithi038_77863_object23129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.956623+0000", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7251356439999999, 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.083029309999999995, 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23130 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956623+0000", 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956623+0000", 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956624+0000", 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956642+0000", 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8856e-05 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956648+0000", 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9540000000000001e-06 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.956654+0000", 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3210000000000004e-06 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035751+0000", 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079097167999999995 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035893+0000", 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014259799999999999 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.036053+0000", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000159543 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037289+0000", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012355739999999999 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037521+0000", 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023217600000000001 2024-03-04T16:05:01.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039578+0000", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020568700000000001 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039620+0000", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2240000000000002e-05 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039652+0000", 2024-03-04T16:05:01.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2382000000000003e-05 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23134 7.6 7:612d0cef:::benchmark_data_smithi038_77863_object23133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.958084+0000", 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7236750299999999, 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.084984606000000004, 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.955 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23134 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958084+0000", 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958084+0000", 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958085+0000", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.595e-06 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958109+0000", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3779999999999999e-05 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958119+0000", 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0015e-05 2024-03-04T16:05:01.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958126+0000", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2899999999999997e-06 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037555+0000", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079428492000000003 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037688+0000", 2024-03-04T16:05:01.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013350600000000001 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.038196+0000", 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00050754999999999995 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.040429+0000", 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022331899999999999 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041898+0000", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014691369999999999 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.042983+0000", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001085031 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.043029+0000", 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5729000000000002e-05 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.043068+0000", 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9291e-05 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:23118 7.6 e102/95 ondisk, result = 0)", 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.958197+0000", 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7235616980000001, 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079042101000000003, 2024-03-04T16:05:01.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958197+0000", 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958197+0000", 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958198+0000", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3510000000000001e-06 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958203+0000", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6349999999999997e-06 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958212+0000", 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0629999999999995e-06 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958219+0000", 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9500000000000004e-06 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037101+0000", 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.078882479000000005 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037108+0000", 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4180000000000002e-06 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037239+0000", 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013120499999999999 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:23125 7.6 e102/95 ondisk, result = 0)", 2024-03-04T16:05:01.965 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.958229+0000", 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7235291190000002, 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079252298999999998, 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958229+0000", 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958229+0000", 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958230+0000", 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4000000000000002e-07 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958233+0000", 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8090000000000001e-06 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.967 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958238+0000", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.322e-06 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958243+0000", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4179999999999999e-06 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037365+0000", 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.079122316999999998 2024-03-04T16:05:01.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037374+0000", 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6379999999999993e-06 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037482+0000", 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108255 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:23136 7.5 e102/95 7:afce9bac:::benchmark_data_smithi038_77863_object23135:head v 102'1442, mlcod=102'1440)", 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.958683+0000", 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7230755250000005, 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077339924000000004, 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958683+0000", 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958683+0000", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958690+0000", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3000000000000004e-06 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958763+0000", 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2624000000000001e-05 2024-03-04T16:05:01.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958772+0000", 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2420000000000006e-06 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958779+0000", 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7680000000000003e-06 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958864+0000", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5288000000000001e-05 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.958897+0000", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3129000000000001e-05 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035984+0000", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077086545000000006 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.036023+0000", 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9028e-05 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23138 7.f 7:f3463d03:::benchmark_data_smithi038_77863_object23137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.959253+0000", 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7225052889999999, 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.082205099000000004, 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.974 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23138 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959253+0000", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959253+0000", 2024-03-04T16:05:01.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959255+0000", 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.578e-06 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959279+0000", 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4592000000000001e-05 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959290+0000", 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0730999999999999e-05 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959298+0000", 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0960000000000001e-06 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959349+0000", 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1184000000000003e-05 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959463+0000", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113715 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959586+0000", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012242199999999999 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035667+0000", 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.076080944999999997 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039907+0000", 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042406830000000003 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.040591+0000", 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068422100000000005 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041405+0000", 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081321599999999996 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041458+0000", 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3715999999999999e-05 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:23135 7.e e102/95 7:739e4d1f:::benchmark_data_smithi038_77863_object23134:head v 102'1469, mlcod=102'1467)", 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.959369+0000", 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7223897929999996, 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.081292018999999993, 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959369+0000", 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959369+0000", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959370+0000", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.189e-06 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959394+0000", 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4332999999999999e-05 2024-03-04T16:05:01.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959404+0000", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9520000000000006e-06 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959411+0000", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6409999999999996e-06 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037311+0000", 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077900369999999997 2024-03-04T16:05:01.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037328+0000", 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6973000000000002e-05 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.040431+0000", 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031025380000000002 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.040661+0000", 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00023002299999999999 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:23129 7.6 e102/95 ondisk, result = 0)", 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.959911+0000", 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7218478409999998, 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077378538999999996, 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:05:01.985 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959911+0000", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959911+0000", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959912+0000", 2024-03-04T16:05:01.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2440000000000001e-06 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959917+0000", 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6759999999999998e-06 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959926+0000", 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2420000000000006e-06 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.959933+0000", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1189999999999999e-06 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037267+0000", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077334165999999996 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037273+0000", 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0419999999999999e-06 2024-03-04T16:05:01.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.037289+0000", 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.605e-05 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:23124 7.f e102/95 ondisk, result = 0)", 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.960362+0000", 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7213967740000005, 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075269637, 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.989 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960362+0000", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960362+0000", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960363+0000", 2024-03-04T16:05:01.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.356e-06 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960368+0000", 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.814e-06 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.034301+0000", 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.073932587999999994 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.034312+0000", 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1566000000000001e-05 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035491+0000", 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011789529999999999 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035500+0000", 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1200000000000008e-06 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035631+0000", 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013124 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:23142 7.f 7:fe8f5ab4:::benchmark_data_smithi038_77863_object23141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:56.960485+0000", 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.7212732559999999, 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.081102627999999996, 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:01.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 23142 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960485+0000", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:01.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960485+0000", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960487+0000", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3239999999999999e-06 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:56.960511+0000", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3873e-05 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:01.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.034383+0000", 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.073872102999999995 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.034394+0000", 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0944000000000001e-05 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035696+0000", 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001302578 2024-03-04T16:05:01.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.035845+0000", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014912200000000001 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.036603+0000", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075729200000000001 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.039874+0000", 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032710230000000001 2024-03-04T16:05:01.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041500+0000", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001626667 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041511+0000", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1223000000000001e-05 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041553+0000", 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1795999999999998e-05 2024-03-04T16:05:01.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.041588+0000", 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4683e-05 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:24069 7.6 7:61283586:::benchmark_data_smithi038_77863_object24068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:57.500183+0000", 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1815760510000004, 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.129172221, 2024-03-04T16:05:01.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 24069 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500183+0000", 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500183+0000", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500184+0000", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4449999999999999e-06 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500205+0000", 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0990999999999999e-05 2024-03-04T16:05:02.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500218+0000", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2758000000000001e-05 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500227+0000", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8170000000000005e-06 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500285+0000", 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8640000000000001e-05 2024-03-04T16:05:02.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500397+0000", 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112255 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.500505+0000", 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107982 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:02.003 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.501678+0000", 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001172197 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.501894+0000", 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000216259 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.628615+0000", 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.12672111899999999 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.629300+0000", 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00068539000000000002 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.629355+0000", 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4367999999999999e-05 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:24073 7.6 7:629e39ab:::benchmark_data_smithi038_77863_object24072:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:57.502895+0000", 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.1788631939999998, 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.12662100600000001, 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 24073 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.006 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502895+0000", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502895+0000", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502897+0000", 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6390000000000001e-06 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502926+0000", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9298999999999998e-05 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502937+0000", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.031e-05 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502944+0000", 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3509999999999996e-06 2024-03-04T16:05:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.502995+0000", 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1060000000000002e-05 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.503122+0000", 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012739199999999999 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:02.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.503225+0000", 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102934 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.504033+0000", 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080796299999999998 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.504567+0000", 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00053385200000000002 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.629426+0000", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.12485835300000001 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.629475+0000", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9644000000000001e-05 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:57.629516+0000", 2024-03-04T16:05:02.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1208999999999998e-05 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:24937 7.f 7:fac5332f:::benchmark_data_smithi038_77863_object24936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:58.012239+0000", 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.669519244, 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.10156525, 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:02.012 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 24937 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012239+0000", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012239+0000", 2024-03-04T16:05:02.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012240+0000", 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3099999999999998e-07 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012261+0000", 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1086000000000001e-05 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012267+0000", 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6790000000000002e-06 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012272+0000", 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0799999999999996e-06 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012355+0000", 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3361000000000004e-05 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012500+0000", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014538999999999999 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.012636+0000", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000135353 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.015498+0000", 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002862458 2024-03-04T16:05:02.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.062953+0000", 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047455019000000001 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.113691+0000", 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.050737906999999999 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.113753+0000", 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.211e-05 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.113805+0000", 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1276000000000002e-05 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:25407 7.7 e103/95 7:ee8d2066:::benchmark_data_smithi038_77863_object25406:head v 103'1568, mlcod=103'1565)", 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:58.311926+0000", 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.3698324230000001, 2024-03-04T16:05:02.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075637843999999996, 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.311926+0000", 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.311926+0000", 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.311927+0000", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0809999999999999e-06 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.312139+0000", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000211296 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.380966+0000", 2024-03-04T16:05:02.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.068827741999999997 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.380981+0000", 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4817000000000001e-05 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.381094+0000", 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112689 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.381113+0000", 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9292000000000001e-05 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.387511+0000", 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0063977799999999996 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:58.387564+0000", 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3146999999999998e-05 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.022 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:26983 7.5 e103/95 7:abf648f5:::benchmark_data_smithi038_77863_object26982:head v 103'1657, mlcod=103'1655)", 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:59.109355+0000", 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5724040700000002, 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075791363, 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109355+0000", 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109355+0000", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109355+0000", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8999999999999996e-07 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109392+0000", 2024-03-04T16:05:02.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6532000000000002e-05 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109399+0000", 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1879999999999996e-06 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109405+0000", 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3759999999999996e-06 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109460+0000", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5282999999999999e-05 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109479+0000", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8539999999999999e-05 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.185117+0000", 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075637773000000005 2024-03-04T16:05:02.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.185146+0000", 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9081000000000002e-05 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:26985 7.5 e103/95 7:ad589788:::benchmark_data_smithi038_77863_object26984:head v 103'1658, mlcod=103'1655)", 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:04:59.109869+0000", 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5718898889999999, 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075435946000000004, 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:02.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109869+0000", 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109869+0000", 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.028 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109869+0000", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4799999999999998e-07 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109931+0000", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2147e-05 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109939+0000", 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3320000000000003e-06 2024-03-04T16:05:02.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.109944+0000", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3890000000000004e-06 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.181875+0000", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.071930583000000006 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.181978+0000", 2024-03-04T16:05:02.030 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102938 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.185278+0000", 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003300632 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:04:59.185305+0000", 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6376999999999998e-05 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:02.031 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:02.032 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:05:02.032 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 14 16 32428 32412 144.679 112.938 0.0128881 0.00690538 2024-03-04T16:05:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.033 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:02.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:02.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:02.165 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:02.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:02.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:02.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:02.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:02.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:02.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:02.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:02.697 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:02 smithi038 ceph-mon[29779]: pgmap v76: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:05:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:02 smithi038 ceph-mon[29779]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:05:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:02 smithi038 ceph-mon[38113]: pgmap v76: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:05:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:02 smithi038 ceph-mon[38113]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:05:02.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:02.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:02.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:02 smithi176 ceph-mon[33619]: pgmap v76: 148 pgs: 148 active+clean; 1.4 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:05:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:02 smithi176 ceph-mon[33619]: osdmap e104: 8 total, 8 up, 8 in 2024-03-04T16:05:02.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:02.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 15 16 34448 34432 143.45 126.25 0.0077172 0.0069652 2024-03-04T16:05:02.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:02.975 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:02.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:02.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:03.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:05:03.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.156 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:03.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:03.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:03.286 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:03.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:03.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:03.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:03.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:03.576 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:03.669 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:03.669 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:03.669 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:05:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:05:03.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:03.724 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:03.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:03.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:03.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:03.930 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 16 16 36287 36271 141.667 114.938 0.00445463 0.00705297 2024-03-04T16:05:03.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:03.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:03.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:04.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:04.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:04.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:04.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:04.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:04.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:04.427 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:04.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:04.573 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:05:04.573 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:05:04.573 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-04T16:05:04.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-04T16:05:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:04.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:04.708 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:04.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:04.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:04.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:04.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:04.931 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 17 16 37843 37827 139.05 97.25 0.00357687 0.00718576 2024-03-04T16:05:04.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.014 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:05.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:05.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:05.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.152 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:05.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:05.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:05.291 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:05.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:05.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:05 smithi176 ceph-mon[33619]: pgmap v78: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-03-04T16:05:05.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:05 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:05.423 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[29779]: pgmap v78: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-03-04T16:05:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[38113]: pgmap v78: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 163 MiB/s wr, 2.60k op/s 2024-03-04T16:05:05.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:05 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:05.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:05.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:05.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:05.689 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:05.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:05.821 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:05.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:05.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:05.931 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 18 16 39526 39510 137.168 105.188 0.00847335 0.0072851 2024-03-04T16:05:05.969 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:05.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:06.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:06.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:06.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:06.256 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:06.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:06.403 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:06.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:06.534 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:06.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.697 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:06.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:06.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:06.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:06.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:06.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 19 16 41127 41111 135.209 100.062 0.0137525 0.00738999 2024-03-04T16:05:06.959 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:06.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:07.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:07.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:07.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:07.219 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:05:07.240 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:07.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:07.255 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:05:07.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:07.368 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:05:07.368 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[38113]: pgmap v79: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[29779]: pgmap v79: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:07.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:07.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:07.408 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:05:07.449 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:05:07.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:07.473 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:05:07.493 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:05:07.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:07.530 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:05:07.535 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:05:07.536 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:05:07.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:07 smithi176 ceph-mon[33619]: pgmap v79: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:05:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:05:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:07.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:07.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:07.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:07.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:07.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:07.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:07.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:05:07.932660+0000 min lat: 0.00146693 max lat: 0.134 avg lat: 0.00752477 2024-03-04T16:05:07.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:05:07.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 20 16 42520 42504 132.801 87.0625 0.00647112 0.00752477 2024-03-04T16:05:08.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:08.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:08.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:08.184 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:08.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[29779]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:05:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[29779]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-04T16:05:08.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1035314254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:08.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[38113]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:05:08.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[38113]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-04T16:05:08.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:08 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1035314254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:08.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.324 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:08.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:08.424 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:08 smithi176 ceph-mon[33619]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-04T16:05:08.425 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:08 smithi176 ceph-mon[33619]: Reconfiguring daemon alertmanager.a on smithi038 2024-03-04T16:05:08.425 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:08 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1035314254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:08.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.466 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:08.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:08.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:08.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:08.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:08.729 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:08.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:08.874 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:08.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:08.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 21 16 44210 44194 131.507 105.625 0.0166583 0.00759841 2024-03-04T16:05:08.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:09.007 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:09.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:09.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:09.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.314 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:09.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:09.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:09.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:09 smithi038 ceph-mon[29779]: pgmap v80: 148 pgs: 148 active+clean; 2.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:05:09.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:09 smithi038 ceph-mon[38113]: pgmap v80: 148 pgs: 148 active+clean; 2.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:05:09.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:09.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:09 smithi176 ceph-mon[33619]: pgmap v80: 148 pgs: 148 active+clean; 2.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:05:09.639 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:09.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:09.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:09.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:09.865 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:09.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:09.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:09.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 22 16 45648 45632 129.614 89.875 0.0131679 0.00770952 2024-03-04T16:05:09.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:10.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:10.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:10.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:10.178 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:05:10.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:10.299 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:10.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/272876987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[29779]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[29779]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-04T16:05:10.378 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/272876987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[38113]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:10 smithi038 ceph-mon[38113]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-04T16:05:10.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:10.431 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:10.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:10.562 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:10 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/272876987' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:10 smithi176 ceph-mon[33619]: from='client.24751 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:05:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:10 smithi176 ceph-mon[33619]: pgmap v81: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-04T16:05:10.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:10.697 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:10.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:10.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:10.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 23 15 47108 47093 127.949 91.3125 0.00650048 0.00781053 2024-03-04T16:05:10.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:10.973 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:10.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:11.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:11.105 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:11.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[29779]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[29779]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[38113]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:05:11.224 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:11 smithi038 ceph-mon[38113]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:05:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.248 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:11.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:11.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:11.396 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:11.499 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:11.530 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:11 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:11 smithi176 ceph-mon[33619]: from='client.24751 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:05:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:11 smithi176 ceph-mon[33619]: osdmap e105: 8 total, 8 up, 8 in 2024-03-04T16:05:11.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:11.661 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:11.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:05:11.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:11.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:11.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:11.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 24 16 48639 48623 126.602 95.625 0.0102325 0.00789272 2024-03-04T16:05:11.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:11.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:12.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:12.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:12.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:12.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:12.335 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:12.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:12.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:12 smithi038 ceph-mon[29779]: pgmap v83: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:12.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:12 smithi038 ceph-mon[38113]: pgmap v83: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:12.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:12.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:12.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:12 smithi176 ceph-mon[33619]: pgmap v83: 148 pgs: 148 active+clean; 2.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:12.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:12.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:12.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:12.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:12.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:12.777 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:12.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:12.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:12.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:12.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:12.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 25 16 50243 50227 125.547 100.25 0.0144267 0.0079589 2024-03-04T16:05:13.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:13.061 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:13.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:13.194 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:13.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:13.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:13.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:13.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:05:13.441 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:13.442 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:05:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:05:13.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:13.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:13.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:13.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:13 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:05:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:13 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:05:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:13.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:13.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.773 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:13.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:13.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:13.922 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:13.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:13.932 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 26 16 51687 51671 124.189 90.25 0.0069369 0.00804723 2024-03-04T16:05:14.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:14.062 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:14.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:14.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:14.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:14.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.350 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:14.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:14.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:14.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:14.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[29779]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:05:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:05:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:05:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[38113]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-04T16:05:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:05:14.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:14 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:05:14.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:14 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:05:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:14 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:05:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:14 smithi176 ceph-mon[33619]: pgmap v84: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 100 MiB/s wr, 1.61k op/s 2024-03-04T16:05:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:14 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:05:14.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:14 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:05:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:14.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:14.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:14.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:14.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:14.878 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:14.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:14.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 27 15 52978 52963 122.58 80.75 0.0492201 0.00815018 2024-03-04T16:05:14.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.018 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:15.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:15.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:15.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:15.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:15.300 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:15.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:15.444 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[29779]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:05:15.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:05:15.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:05:15.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:05:15.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:05:15.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:15 smithi038 ceph-mon[38113]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:05:15.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.579 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:15.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:15 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:15 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:15 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:15 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:05:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:15 smithi176 ceph-mon[33619]: osdmap e106: 8 total, 8 up, 8 in 2024-03-04T16:05:15.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:15.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:15.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:15.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:15.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 28 16 54262 54246 121.066 80.1875 0.0035773 0.0082541 2024-03-04T16:05:15.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:15.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:15.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:16.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:16.119 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:16.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:16.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:16.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:16.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:16.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:05:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[29779]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[38113]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:05:16.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:16 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:05:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:16.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:16 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:05:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:16 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:05:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:16 smithi176 ceph-mon[33619]: pgmap v86: 148 pgs: 148 active+clean; 3.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-04T16:05:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:16 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:05:16.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:16 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:05:16.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:16.671 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:16.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:16.804 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:16.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:16.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 29 15 55571 55556 119.714 81.875 0.0115976 0.00834868 2024-03-04T16:05:16.947 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:16.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:16.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:17.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:17.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:17.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:17.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:17.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:17.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:17.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:17.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:17.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:05:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:05:17.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:17 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:05:17.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:05:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:05:17.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:05:17.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:05:17.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:05:17.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:17 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:05:17.619 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:17.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:17.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:17.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:17.891 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:17.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:17.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 30 16 56728 56712 118.132 72.25 0.00499897 0.00846023 2024-03-04T16:05:17.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:18.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:18.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:18.164 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:18.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:18.295 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:18.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:18.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:18.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:05:18.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:05:18.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:05:18.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:05:18.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-04T16:05:18.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:05:18.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:18 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:05:18.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:18.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:05:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:05:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:05:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:05:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-04T16:05:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:05:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:18 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:05:18.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.708 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:18.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.840 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:18.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 31 15 58147 58132 117.184 88.75 0.00490266 0.00852817 2024-03-04T16:05:18.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:18.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:18.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:19.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:19.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:19.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:19.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:19.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:05:19.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:05:19.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:05:19.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:05:19.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[29779]: osdmap e107: 8 total, 8 up, 8 in 2024-03-04T16:05:19.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:05:19.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:05:19.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:05:19.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:05:19.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:19 smithi038 ceph-mon[38113]: osdmap e107: 8 total, 8 up, 8 in 2024-03-04T16:05:19.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:19.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:19.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:19.543 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:19 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:05:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:19 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:05:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:19 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:05:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:19 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:05:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:19 smithi176 ceph-mon[33619]: osdmap e107: 8 total, 8 up, 8 in 2024-03-04T16:05:19.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:19.654 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:05:19.654 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:05:19.675 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:19.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:19.805 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:19.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:19.933 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 32 16 59455 59439 116.074 81.6875 0.00968681 0.00860946 2024-03-04T16:05:19.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:19.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:20.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:20.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:20.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:20.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:20.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:20.358 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:20.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:20.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:20.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: pgmap v89: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:05:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:05:20.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:20 smithi176 ceph-mon[33619]: osdmap e108: 8 total, 8 up, 8 in 2024-03-04T16:05:20.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:05:20.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:05:20.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:05:20.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: pgmap v89: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[29779]: osdmap e108: 8 total, 8 up, 8 in 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: pgmap v89: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:05:20.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:05:20.629 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:05:20.629 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:20 smithi038 ceph-mon[38113]: osdmap e108: 8 total, 8 up, 8 in 2024-03-04T16:05:20.637 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:20.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:20.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:20.862 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:05:20.862 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:05:20.862 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:05:20.862 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:05:20.862 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:05:20.863 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:05:20.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:20.932 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:20.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:20.934 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 33 16 60883 60867 115.26 89.25 0.0179685 0.0086712 2024-03-04T16:05:20.994 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:20 smithi038 systemd[1]: Stopping Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:21.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:21.069 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:21.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:21.200 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:21.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:21.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:21.437 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:21.467 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:21.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:05:21.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:05:21.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:05:21.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:05:21.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:05:21.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:21 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:05:21.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:05:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:05:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:05:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:05:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:05:21.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:05:21.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:05:21.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:21 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:05:21.614 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:21.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:21.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:21.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:21.899 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:21.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:21.936 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 34 16 62274 62258 114.427 86.9375 0.00878067 0.00873444 2024-03-04T16:05:22.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:22.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:22.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:22.185 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:22.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:22.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:22.316 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:22.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:22.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:05:22.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:05:22.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 824 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:05:22.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:05:22.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 824 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:05:22.443 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:05:22.444 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:05:22.444 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:05:22.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:05:22.445 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:22.461 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:22.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:05:22.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:22.589 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:22.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:22.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:05:22.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: pgmap v91: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 824 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:05:22.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:05:22.647 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:05:22.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-04T16:05:22.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:22.724 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:22.724 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:22.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:22.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:22.870 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:22.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:22.935 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 35 16 63485 63469 113.316 75.6875 0.0139125 0.00881899 2024-03-04T16:05:22.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:22.993 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:22 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[73283]: ts=2024-03-04T16:05:22.916Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:05:23.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:23.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.151 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:23.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:23.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:23.282 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:23.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:23.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:05:23.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:05:23.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:23.288 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:05:23.288 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:23 smithi038 podman[81611]: 2024-03-04 16:05:23.038582153 +0000 UTC m=+0.318918891 container died 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:23.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:23.437 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:23.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:23.568 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:23.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:05:23.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:05:23.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:05:23.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:23.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:05:23.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:05:23.592 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:05:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:05:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:23 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:05:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:23 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:05:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:23 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:05:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:23 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:05:23.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:23.697 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:23.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.842 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:23.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:23.843 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:23 smithi038 podman[81611]: 2024-03-04 16:05:23.661858988 +0000 UTC m=+0.942195721 container cleanup 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:23.843 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:23 smithi038 bash[81611]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T16:05:23.935 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 36 16 64799 64783 112.449 82.125 0.0207064 0.00888781 2024-03-04T16:05:23.944 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:23.976 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:23.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:23.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:24.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.112 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:24.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:24.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:24.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:24.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:24.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:05:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: pgmap v92: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:05:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: pgmap v92: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:05:24.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:24 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:05:24.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:24.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:05:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: pgmap v92: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:05:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:24 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:05:24.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:24.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:24.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:24.825 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:24.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:24.935 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 37 16 66107 66091 111.619 81.75 0.0132687 0.00895359 2024-03-04T16:05:24.956 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:24.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:25.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:25.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:25.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:25.216 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:25.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:25.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:25.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:05:25.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:05:25.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:05:25.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:05:25.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:05:25.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:05:25.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[29779]: osdmap e109: 8 total, 8 up, 8 in 2024-03-04T16:05:25.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:05:25.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:05:25.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:05:25.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:05:25.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:05:25.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:05:25.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:25 smithi038 ceph-mon[38113]: osdmap e109: 8 total, 8 up, 8 in 2024-03-04T16:05:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:25.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:05:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:05:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:05:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:05:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:05:25.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:05:25.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:25 smithi176 ceph-mon[33619]: osdmap e109: 8 total, 8 up, 8 in 2024-03-04T16:05:25.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:25.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:25.754 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:25 smithi038 podman[81668]: 2024-03-04 16:05:25.49403502 +0000 UTC m=+2.528378315 container remove 246cf6ba198e215bdd46e1278e5ba7ae52723adc98e7b937b702febd3435391c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:25.755 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:25 smithi038 podman[81668]: 2024-03-04 16:05:25.595523062 +0000 UTC m=+2.629866362 volume remove f7ebc9ceaa3ed1aa49840b3915505a87f5ba644779c35eb3c09b040830d86b55 2024-03-04T16:05:25.773 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:25.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:25.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:25.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:25.940 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 38 16 67348 67332 110.71 77.5625 0.0258725 0.0090274 2024-03-04T16:05:26.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:26.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:26.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:26.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:26.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:26.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:26.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 724 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:05:26.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:05:26.488 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:05:26.488 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:05:26.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:05:26.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:05:26.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 724 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-04T16:05:26.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:05:26.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:26 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:05:26.514 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:05:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:05:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:05:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:05:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: pgmap v94: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 724 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-04T16:05:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:05:26.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:26 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:05:26.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.652 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:26.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:26.736 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:05:26.743 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:26 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T16:05:26.743 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:26 smithi038 systemd[1]: Stopped Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:26.743 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:26 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Consumed 1.184s CPU time. 2024-03-04T16:05:26.743 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:26 smithi038 systemd[1]: Starting Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:26.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:26.777 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:05:26.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:26.820 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:05:26.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:05:26.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:26.902 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:05:26.913 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:26.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:26.940 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 39 16 68579 68563 109.844 76.9375 0.0115626 0.00909847 2024-03-04T16:05:26.943 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:05:26.989 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:05:27.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:27.030 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:05:27.034 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:05:27.035 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:05:27.061 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:27.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:27.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:27.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:27.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:27.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:27.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.583 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4212154531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:05:27.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:27 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:05:27.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:27.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:05:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:05:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:05:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:05:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4212154531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:05:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:05:27.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4212154531' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:27.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:05:27.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:27 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:05:27.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:27.852 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:27.940 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:05:27.940927+0000 min lat: 0.00146693 max lat: 0.134 avg lat: 0.00919371 2024-03-04T16:05:27.941 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:05:27.941 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 40 16 69586 69570 108.671 62.9375 0.00380514 0.00919371 2024-03-04T16:05:27.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:27.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:27.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:28.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:28.118 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:28.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:28.258 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:28.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:28.388 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:28.490 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:28.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:05:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:05:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:05:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:05:28.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:05:28.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:05:28.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:05:28.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:05:28.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:28 smithi176 ceph-mon[33619]: pgmap v95: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:28.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:28.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:05:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:05:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[29779]: pgmap v95: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:05:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:05:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:05:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:05:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:05:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:28 smithi038 ceph-mon[38113]: pgmap v95: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:28.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:28.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:28.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:28.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:28.940 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 41 16 70845 70829 107.939 78.6875 0.00763478 0.00925898 2024-03-04T16:05:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:28.959 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:28.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:29.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:29.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:29.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:29.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:29.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:29.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:29.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:29.494 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:05:29.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:05:29.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:05:29.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:05:29.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:29 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:05:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:29.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:05:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:29 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:05:29.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:29.674 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:29.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:29.812 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:29.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:29.941 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 42 16 72065 72049 107.185 76.25 0.011505 0.0093238 2024-03-04T16:05:29.955 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:29.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:30.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:30.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:30.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:30.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:30.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:30.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.386 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:30.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:30.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:05:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[29779]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-04T16:05:30.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:05:30.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:30 smithi038 ceph-mon[38113]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:05:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:30.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:05:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:05:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:05:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:05:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:05:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:05:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: pgmap v96: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-04T16:05:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: osdmap e110: 8 total, 8 up, 8 in 2024-03-04T16:05:30.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:30 smithi176 ceph-mon[33619]: osdmap e111: 8 total, 8 up, 8 in 2024-03-04T16:05:30.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:30.662 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:30.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:30.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:30.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:30.941 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 43 16 73300 73284 106.486 77.1875 0.00747338 0.00938578 2024-03-04T16:05:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:30.950 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:30.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:31.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:31.081 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:31.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.222 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:31.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:31.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:31.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:31.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:31.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:31 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:05:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:31 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:05:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:31 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:05:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:31 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:05:31.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:31.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:05:31.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:05:31.694 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:31 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:05:31.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.787 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:31.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:31.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:31.916 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:31.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:31.941 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 44 16 74508 74492 105.781 75.5 0.0168201 0.00944807 2024-03-04T16:05:32.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:32.043 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:32.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:32.198 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:32.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:32.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:32.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:32.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:32.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:32.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: pgmap v99: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 825 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:05:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:32 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:05:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: pgmap v99: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 825 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:05:32.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: pgmap v99: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 825 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:05:32.642 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:32 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:05:32.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:32.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:32.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:32.904 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:32 smithi038 podman[82663]: 2024-03-04 16:05:32.650863902 +0000 UTC m=+0.101859461 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-04T16:05:32.904 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:32 smithi038 podman[82663]: 2024-03-04 16:05:32.903592201 +0000 UTC m=+0.354587787 volume create b64903e08967d3d9b0b7b0f6d1f7d1fe1f6ea2ba51e64e8fa6331333be2e6b90 2024-03-04T16:05:32.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:32.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:32.942 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 45 16 75558 75542 104.889 65.625 0.0244928 0.00952885 2024-03-04T16:05:33.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:33.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.055 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:33.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:33.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:33.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:33.224 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:33 smithi038 podman[82663]: 2024-03-04 16:05:33.004476576 +0000 UTC m=+0.455472179 container create 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:33.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:33.324 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:05:33.324 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:05:33.324 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:05:33.325 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:05:33.325 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:05:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:33.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:05:33.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:33.475 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:33.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:05:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:05:33.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:05:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:05:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:33 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:05:33.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.608 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:33.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:33.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:33.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:33.845 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:05:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:33 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:05:33.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:33.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:33.942 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 46 16 76673 76657 104.123 69.6875 0.0179917 0.00959898 2024-03-04T16:05:33.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:34.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:34.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:34.145 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:34.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:34.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:34.303 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 podman[82663]: 2024-03-04 16:05:34.105742641 +0000 UTC m=+1.556738201 container init 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 podman[82663]: 2024-03-04 16:05:34.112768721 +0000 UTC m=+1.563764275 container start 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.133Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.134Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.135Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.38 port=9094 2024-03-04T16:05:34.375 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.138Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.190Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.191Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.195Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:34.195Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 bash[82663]: 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 2024-03-04T16:05:34.376 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:34 smithi038 systemd[1]: Started Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:34.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:34.447 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:34.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.585 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:34.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:34.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:34.717 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:05:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:05:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:05:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:05:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: osdmap e112: 8 total, 8 up, 8 in 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: osdmap e112: 8 total, 8 up, 8 in 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:34 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:34.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:34.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: pgmap v100: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1515212169' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: osdmap e112: 8 total, 8 up, 8 in 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:05:34.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:05:34.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:05:34.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:05:34.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:05:34.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:34 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:34.943 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 47 16 77801 77785 103.406 70.5 0.011073 0.00966605 2024-03-04T16:05:34.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:34.996 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:34.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:35.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:35.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:35.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.281 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:35.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:35.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:35.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:35.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:35.562 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:35.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:35.696 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:35.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:35.836 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:35.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:35.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:35.959 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 48 16 79008 78992 102.789 75.4375 0.0101008 0.0097227 2024-03-04T16:05:35.974 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:35.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:35.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:36.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:36.121 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:36.170 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:36.139Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001113413s 2024-03-04T16:05:36.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:05:36.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: Reconfiguring daemon prometheus.a on smithi176 2024-03-04T16:05:36.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:05:36.170 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: osdmap e113: 8 total, 8 up, 8 in 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 5.d deep-scrub starts 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[29779]: 5.d deep-scrub ok 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: Reconfiguring daemon prometheus.a on smithi176 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: osdmap e113: 8 total, 8 up, 8 in 2024-03-04T16:05:36.171 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:05:36.172 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 5.d deep-scrub starts 2024-03-04T16:05:36.172 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:36 smithi038 ceph-mon[38113]: 5.d deep-scrub ok 2024-03-04T16:05:36.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:36.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:05:36.326 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:05:36.327 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:05:36.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:36.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:36.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:36.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-04T16:05:36.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: Reconfiguring daemon prometheus.a on smithi176 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: osdmap e113: 8 total, 8 up, 8 in 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 5.d deep-scrub starts 2024-03-04T16:05:36.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:36 smithi176 ceph-mon[33619]: 5.d deep-scrub ok 2024-03-04T16:05:36.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:36.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:36.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:36.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:36.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:36.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:36.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:36.959 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 49 16 80180 80164 102.186 73.25 0.00916123 0.00978144 2024-03-04T16:05:37.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:37.062 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:37.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:37.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:37.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:37.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:37.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 856 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:37.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:37.324 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 856 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-04T16:05:37.324 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:05:37.324 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:05:37.324 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:05:37.324 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:05:37.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:05:37.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:05:37.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:37.325 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:37.336 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:37.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:05:37.379 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:05:37.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-04T16:05:37.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:05:37.427 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:37.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.529 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:37.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: pgmap v103: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 856 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-04T16:05:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:05:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:05:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:05:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:05:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:05:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:05:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:37.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:05:37.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:37.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:37.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:37.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:37.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:37.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:37.960 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 50 16 81132 81116 101.333 59.5 0.0189202 0.00986279 2024-03-04T16:05:37.970 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:37.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:38.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:38.102 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:38.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:38.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:38.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.11 deep-scrub starts 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.11 deep-scrub ok 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:05:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:05:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:05:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:05:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: pgmap v104: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:05:38.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:38 smithi176 ceph-mon[33619]: 7.8 scrub starts 2024-03-04T16:05:38.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:38.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.11 deep-scrub starts 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:05:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.11 deep-scrub ok 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: pgmap v104: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[29779]: 7.8 scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.11 deep-scrub starts 2024-03-04T16:05:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.11 deep-scrub ok 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: pgmap v104: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:05:38.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:38 smithi038 ceph-mon[38113]: 7.8 scrub starts 2024-03-04T16:05:38.515 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:38.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:38.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:38.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:38.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:38.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:38.960 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 51 16 82184 82168 100.635 65.75 0.0116414 0.0099309 2024-03-04T16:05:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:38.987 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:38.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:39.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:39.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:39.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:39.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.256 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:39.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:39.276 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:38 smithi176 systemd[1]: Stopping Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:39.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-mon[33619]: 3.5 deep-scrub starts 2024-03-04T16:05:39.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-mon[33619]: 3.5 deep-scrub ok 2024-03-04T16:05:39.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:05:39.277 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:05:39.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:39.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[29779]: 3.5 deep-scrub starts 2024-03-04T16:05:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[29779]: 3.5 deep-scrub ok 2024-03-04T16:05:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:05:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:05:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[38113]: 3.5 deep-scrub starts 2024-03-04T16:05:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[38113]: 3.5 deep-scrub ok 2024-03-04T16:05:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:05:39.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:39 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:05:39.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:39.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.275Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:05:39.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-04T16:05:39.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[62025]: ts=2024-03-04T16:05:39.276Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-04T16:05:39.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 podman[69218]: 2024-03-04 16:05:39.286279678 +0000 UTC m=+0.135080321 container died 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:39.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:39.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:39.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:39.818 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:39.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:39.947 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 podman[69218]: 2024-03-04 16:05:39.701560451 +0000 UTC m=+0.550361086 container cleanup 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:39.947 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 bash[69218]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:05:39.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:39.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:39.959 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 52 16 83255 83239 99.9875 66.9375 0.0116187 0.00999604 2024-03-04T16:05:40.052 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:40.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:40.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:40.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:40.218 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:40.253 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:39 smithi176 podman[69247]: 2024-03-04 16:05:39.946590026 +0000 UTC m=+0.659885816 container remove 4c1c47d03102a9a2538fa1cb7ea485e5123d7e2420fb9a919642593415c14f45 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:40.253 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:05:40.253 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:05:40.253 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: 7.8 scrub ok 2024-03-04T16:05:40.253 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: pgmap v105: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:40.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:05:40.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:40 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:05:40.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:40.345 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:40.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: 7.8 scrub ok 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: pgmap v105: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:05:40.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: 7.8 scrub ok 2024-03-04T16:05:40.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: pgmap v105: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-04T16:05:40.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:05:40.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:40 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:05:40.492 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:40.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:40.608 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-04T16:05:40.608 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 systemd[1]: Stopped Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:40.608 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@prometheus.a.service: Consumed 1.359s CPU time. 2024-03-04T16:05:40.608 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 systemd[1]: Starting Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:05:40.608 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 podman[69440]: 2024-03-04 16:05:40.39633389 +0000 UTC m=+0.023241135 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-04T16:05:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.640 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:40.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:40.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:40.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:40.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:40.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:40.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:40.959 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 53 16 84132 84116 99.1355 54.8125 0.0214009 0.010081 2024-03-04T16:05:41.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.046 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:41.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:41.108 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:40 smithi176 podman[69440]: 2024-03-04 16:05:40.763254954 +0000 UTC m=+0.390162196 container create 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:41.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.176 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:41.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:41.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:41.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:41.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:41.432 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:05:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:05:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:05:41.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:41.580 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:05:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:05:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:05:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:05:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:05:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 podman[69440]: 2024-03-04 16:05:41.316035507 +0000 UTC m=+0.942942748 container init 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 podman[69440]: 2024-03-04 16:05:41.318555219 +0000 UTC m=+0.945462465 container start 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.342Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.342Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.342Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi176 (none))" 2024-03-04T16:05:41.609 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.342Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.342Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.344Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.344Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.345Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.345Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.348Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.348Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.701µs 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.348Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.348Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.349Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.349Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.592µs wal_replay_duration=881.422µs wbl_replay_duration=89ns total_replay_duration=917.858µs 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.349Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.349Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-04T16:05:41.610 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.349Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-04T16:05:41.611 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.380Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=30.175632ms db_storage=910ns remote_storage=1.527µs web_handler=520ns query_engine=620ns scrape=536.487µs scrape_sd=279.692µs notify=16.327µs notify_sd=19.257µs rules=29.056066ms tracing=7.599µs 2024-03-04T16:05:41.611 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.380Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-04T16:05:41.611 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:05:41.380Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-04T16:05:41.611 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 bash[69440]: 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 2024-03-04T16:05:41.611 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:05:41 smithi176 systemd[1]: Started Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:05:41.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.711 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:41.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:41.775 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPING 2024-03-04T16:05:41.775 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPED 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTING 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Serving on http://:::9283 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTED 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPING 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPED 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTING 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Serving on http://:::9283 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTED 2024-03-04T16:05:41.776 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPING 2024-03-04T16:05:41.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:41.833 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:41.853 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:05:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:41.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:41.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:41.904 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:05:41.953 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:05:41.961 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 54 16 84783 84767 98.0522 40.6875 0.0440013 0.0101931 2024-03-04T16:05:41.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:41.995 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:05:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.000 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:42.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:42.039 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:05:42.040 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:05:42.040 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STOPPED 2024-03-04T16:05:42.040 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTING 2024-03-04T16:05:42.041 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Serving on http://:::9283 2024-03-04T16:05:42.041 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:05:41] ENGINE Bus STARTED 2024-03-04T16:05:42.081 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:05:42.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:42.129 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:05:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.138 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:42.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:42.179 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:05:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:05:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:05:42.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:42.274 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:42.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:05:42.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 4.e deep-scrub starts 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: 4.e deep-scrub ok 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: osdmap e114: 8 total, 8 up, 8 in 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 644 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.295 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.296 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2934584796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 4.e deep-scrub starts 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: 4.e deep-scrub ok 2024-03-04T16:05:42.297 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: osdmap e114: 8 total, 8 up, 8 in 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 644 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.299 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.299 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:42.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:42 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2934584796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:42.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.419 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:42.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:42.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:42.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:42.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 4.e deep-scrub starts 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: 4.e deep-scrub ok 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: osdmap e114: 8 total, 8 up, 8 in 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: pgmap v107: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 644 B/s rd, 45 MiB/s wr, 717 op/s 2024-03-04T16:05:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi038.front.sepia.ceph.com:9093"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-04T16:05:42.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi176.front.sepia.ceph.com:9095"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mon.? v1:172.21.15.38:0/4135988966' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi176.front.sepia.ceph.com:3000"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:05:42.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:42 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2934584796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:42.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.691 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:42.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:42.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:42.829 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:42.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:42.961 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 55 16 85747 85731 97.3634 60.25 0.0236104 0.0102654 2024-03-04T16:05:42.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:42.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:43.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:43.120 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:43.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:43.253 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:43.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:43.389 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:43.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:05:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:05:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:05:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:05:43.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.10 deep-scrub starts 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[29779]: 6.10 deep-scrub ok 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:05:43.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.10 deep-scrub starts 2024-03-04T16:05:43.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:43 smithi038 ceph-mon[38113]: 6.10 deep-scrub ok 2024-03-04T16:05:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.526 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:43.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:05:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:05:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:05:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:05:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:05:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:05:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.10 deep-scrub starts 2024-03-04T16:05:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:43 smithi176 ceph-mon[33619]: 6.10 deep-scrub ok 2024-03-04T16:05:43.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:43.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:43.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.812 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:43.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:43.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:43.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:43.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:43.961 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 56 16 86848 86832 96.8537 68.8125 0.00683793 0.0103199 2024-03-04T16:05:44.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:44.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:44.170 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:05:44.142Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004188547s 2024-03-04T16:05:44.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:44.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:44.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:44 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:05:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:44 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:05:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:44 smithi176 ceph-mon[33619]: pgmap v108: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:05:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:44 smithi176 ceph-mon[33619]: 4.b deep-scrub starts 2024-03-04T16:05:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:44 smithi176 ceph-mon[33619]: 4.b deep-scrub ok 2024-03-04T16:05:44.390 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:44.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[29779]: pgmap v108: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[29779]: 4.b deep-scrub starts 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[29779]: 4.b deep-scrub ok 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:05:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[38113]: pgmap v108: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:05:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[38113]: 4.b deep-scrub starts 2024-03-04T16:05:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:44 smithi038 ceph-mon[38113]: 4.b deep-scrub ok 2024-03-04T16:05:44.529 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:44.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:44.668 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:44.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:44.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:44.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:44.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:44.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:44.961 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 57 16 87980 87964 96.3959 70.75 0.00946204 0.0103691 2024-03-04T16:05:45.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:45.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:45.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:05:45.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:45.212 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:45.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:45.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:45.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:05:45.435 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:05:45.435 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[29779]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:05:45.435 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:05:45.435 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:05:45.435 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:45 smithi038 ceph-mon[38113]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:05:45.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.513 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:45.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:45 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:45 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:05:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:45 smithi176 ceph-mon[33619]: osdmap e115: 8 total, 8 up, 8 in 2024-03-04T16:05:45.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:45.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:45.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:45.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:45.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:45.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:45.933 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:45.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:45.961 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 58 16 89008 88992 95.8418 64.25 0.0116171 0.0104288 2024-03-04T16:05:46.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.083 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:46.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:46.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:46.221 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:46.226 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:05:46.226 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:05:46.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:46.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:46.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:46.509 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:46.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:46 smithi176 ceph-mon[33619]: 7.1 scrub starts 2024-03-04T16:05:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:46 smithi176 ceph-mon[33619]: pgmap v110: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-04T16:05:46.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:46.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:46 smithi038 ceph-mon[29779]: 7.1 scrub starts 2024-03-04T16:05:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:46 smithi038 ceph-mon[29779]: pgmap v110: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-04T16:05:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:46 smithi038 ceph-mon[38113]: 7.1 scrub starts 2024-03-04T16:05:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:46 smithi038 ceph-mon[38113]: pgmap v110: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 60 MiB/s wr, 962 op/s 2024-03-04T16:05:46.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:46.788 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:46.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:46.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:46.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:46.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 59 16 90020 90004 95.2896 63.25 0.0844059 0.010483 2024-03-04T16:05:47.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:47.060 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:47.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.198 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:47.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:47.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:47.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:47.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:47.510 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:47.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:47 smithi176 ceph-mon[33619]: 6.1b deep-scrub starts 2024-03-04T16:05:47.540 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:47 smithi176 ceph-mon[33619]: 6.1b deep-scrub ok 2024-03-04T16:05:47.540 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:47 smithi176 ceph-mon[33619]: 7.e deep-scrub starts 2024-03-04T16:05:47.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:47.653 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[29779]: 6.1b deep-scrub starts 2024-03-04T16:05:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[29779]: 6.1b deep-scrub ok 2024-03-04T16:05:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[29779]: 7.e deep-scrub starts 2024-03-04T16:05:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[38113]: 6.1b deep-scrub starts 2024-03-04T16:05:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[38113]: 6.1b deep-scrub ok 2024-03-04T16:05:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:47 smithi038 ceph-mon[38113]: 7.e deep-scrub starts 2024-03-04T16:05:47.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:47.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:47.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:47.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:47.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:47.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:05:47.962959+0000 min lat: 0.00146693 max lat: 0.166427 avg lat: 0.0105601 2024-03-04T16:05:47.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:05:47.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 60 16 90918 90902 94.6362 56.125 0.012784 0.0105601 2024-03-04T16:05:48.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:48.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:48.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:48.241 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:48.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.385 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:48.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:48.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: 7.1 scrub ok 2024-03-04T16:05:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: 7.d scrub starts 2024-03-04T16:05:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: pgmap v111: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 809 op/s 2024-03-04T16:05:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:05:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:05:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: 7.1 scrub ok 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: 7.d scrub starts 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: pgmap v111: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 809 op/s 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:48 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.520 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:48.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:05:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: 7.1 scrub ok 2024-03-04T16:05:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: 7.d scrub starts 2024-03-04T16:05:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: pgmap v111: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 809 op/s 2024-03-04T16:05:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:05:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:05:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:48 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:48.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:48.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:48.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.844 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:48.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:48.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:48.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 61 16 91780 91764 93.9682 53.875 0.00338746 0.0106311 2024-03-04T16:05:48.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:48.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:49.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:49.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:49.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:49.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:49.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:49.360 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:49.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:49.510 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:05:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:05:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 7.d scrub ok 2024-03-04T16:05:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:05:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:05:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:05:49.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:49 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:05:49.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:49.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 7.d scrub ok 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:05:49.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 7.d scrub ok 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:05:49.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:49 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:05:49.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:49.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:49.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:49.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:49.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 62 15 92536 92521 93.216 47.3125 0.00528738 0.0107203 2024-03-04T16:05:49.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:50.036 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:50.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:50.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:50.247 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:05:50.247 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:05:50.247 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:05:50.247 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:05:50.247 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:05:50.248 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:05:50.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:50.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:50.324 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:05:50.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:50.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:50.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:50.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: pgmap v112: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2495859558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:05:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:50 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:05:50.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:50.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: pgmap v112: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-04T16:05:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2495859558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:05:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: pgmap v112: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2495859558' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: from='client.24781 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:05:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:50 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:05:50.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:50.826 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:50.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:50.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:50.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:50.962 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 63 16 93525 93509 92.7167 61.75 0.0171972 0.0107785 2024-03-04T16:05:51.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:51.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:51.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:51.219 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:51.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:51.350 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:51.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:51.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:51.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 5.10 deep-scrub starts 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 5.10 deep-scrub ok 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 4.2 deep-scrub starts 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: 4.2 deep-scrub ok 2024-03-04T16:05:51.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[29779]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 5.10 deep-scrub starts 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 5.10 deep-scrub ok 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 4.2 deep-scrub starts 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: 4.2 deep-scrub ok 2024-03-04T16:05:51.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:51.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:05:51.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:51 smithi038 ceph-mon[38113]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:05:51.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 5.10 deep-scrub starts 2024-03-04T16:05:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 5.10 deep-scrub ok 2024-03-04T16:05:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:05:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:05:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 4.2 deep-scrub starts 2024-03-04T16:05:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: 4.2 deep-scrub ok 2024-03-04T16:05:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:05:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: from='client.24781 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:05:51.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:51 smithi176 ceph-mon[33619]: osdmap e116: 8 total, 8 up, 8 in 2024-03-04T16:05:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:51.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:51.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:51.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:05:51.848 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:05:51.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-04T16:05:51.852 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:51.864 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:51.868 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:51.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:51.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:51.964 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 64 16 94415 94399 92.1349 55.625 0.0128448 0.0108459 2024-03-04T16:05:51.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:52.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:52.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:52.200 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:52.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:52.347 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:52.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.498 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:52.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[29779]: pgmap v114: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 662 op/s 2024-03-04T16:05:52.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:52.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:52.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:52.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:52.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:52 smithi176 ceph-mon[33619]: pgmap v114: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 662 op/s 2024-03-04T16:05:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:52.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:05:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:05:52.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:52.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:52.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:52.964 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 65 15 95315 95300 91.584 56.3125 0.0109052 0.0109114 2024-03-04T16:05:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[38113]: pgmap v114: 148 pgs: 1 active+clean+scrubbing+deep, 146 active+clean, 1 active+clean+scrubbing; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 662 op/s 2024-03-04T16:05:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:05:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:05:53.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:53.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:53.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:53.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.190 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:53.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:53.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:05:53.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:53.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:53.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:53.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.493 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:53.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:53.496 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:05:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:05:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:05:53.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:53.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:53.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:53.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:53.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:53.858 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:53.859 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:05:53.876 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:53.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:53.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:53.965 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 66 16 96179 96163 91.0138 53.9375 0.00849855 0.0109779 2024-03-04T16:05:54.007 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.037 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:54.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:54.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.179 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:54.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:54.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:54.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:54.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:54.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:54.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:54.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:54 smithi176 ceph-mon[33619]: pgmap v115: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:54.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:54.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:54.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:54.935 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:54 smithi038 ceph-mon[29779]: pgmap v115: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:54.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:54 smithi038 ceph-mon[38113]: pgmap v115: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-04T16:05:54.964 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 67 16 97079 97063 90.4951 56.25 0.0252505 0.011044 2024-03-04T16:05:55.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:55.052 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:55.066 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:55.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:55.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:05:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.210 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:55.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:55.221 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:05:55.266 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:05:55.309 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:05:55.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:55.351 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:05:55.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:55.396 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:05:55.439 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:05:55.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:55.483 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:05:55.494 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:05:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:05:55.495 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:05:55.497 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:55.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:55.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:55.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:55.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/337441130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:05:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/337441130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.759 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:55.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:05:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:05:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/337441130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:05:55.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:55.914 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:55.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:55.965 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 68 16 97870 97854 89.8915 49.4375 0.00503653 0.0111183 2024-03-04T16:05:56.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:56.044 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:56.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:56.194 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:56.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.338 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:56.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:56.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:56.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:56.584 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:05:56.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:05:56.715 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:56 smithi038 ceph-mon[29779]: pgmap v116: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-04T16:05:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:56 smithi038 ceph-mon[29779]: osdmap e117: 8 total, 8 up, 8 in 2024-03-04T16:05:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:56 smithi038 ceph-mon[38113]: pgmap v116: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-04T16:05:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:56 smithi038 ceph-mon[38113]: osdmap e117: 8 total, 8 up, 8 in 2024-03-04T16:05:56.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:56.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:56.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:56 smithi176 ceph-mon[33619]: pgmap v116: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1022 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-04T16:05:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:56 smithi176 ceph-mon[33619]: osdmap e117: 8 total, 8 up, 8 in 2024-03-04T16:05:56.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:56.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:56.965 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 69 16 98780 98764 89.4131 56.875 0.00640367 0.0111759 2024-03-04T16:05:56.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:57.035 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:57.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:57.165 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:57.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:05:57.327 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:05:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:05:57.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:57.476 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:57.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:57.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:57.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:57.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:57.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:57.911 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:57.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:57.969 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 70 16 99681 99665 88.935 56.3125 0.0128785 0.011233 2024-03-04T16:05:58.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:58.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:58.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:58.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:58.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:58.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:58.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:58.433 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:05:58.433 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:05:58.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.433 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:99979 7.e 7:7841db3a:::benchmark_data_smithi038_77863_object99978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.317260+0000", 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.088416619000000002, 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.088461247000000007, 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 99979 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317260+0000", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317260+0000", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317262+0000", 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1220000000000002e-06 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317296+0000", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3402999999999999e-05 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317310+0000", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4633e-05 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.317318+0000", 2024-03-04T16:05:58.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.374e-06 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.319354+0000", 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020366450000000001 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.319481+0000", 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000126353 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.319637+0000", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000156398 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.322995+0000", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033575300000000001 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.323346+0000", 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035157800000000001 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:99979 7.e e117/95)", 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.405019+0000", 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00065786800000000004, 2024-03-04T16:05:58.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079815300000000001, 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405019+0000", 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405019+0000", 2024-03-04T16:05:58.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405021+0000", 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.519e-06 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405027+0000", 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405042+0000", 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5291999999999999e-05 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405050+0000", 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5969999999999992e-06 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "rep_scrubmap(7.e e95 from shard 0)", 2024-03-04T16:05:58.442 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.405073+0000", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00060348400000000005, 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000791955, 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405073+0000", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405073+0000", 2024-03-04T16:05:58.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405074+0000", 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5799999999999999e-07 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405089+0000", 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.504e-05 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:58.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405096+0000", 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9029999999999998e-06 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405102+0000", 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8139999999999997e-06 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:100044 7.3 e117/95 7:c384f0be:::benchmark_data_smithi038_77863_object100043:head v 117'6396, mlcod=117'6394)", 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.405135+0000", 2024-03-04T16:05:58.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00054215000000000003, 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00078263599999999997, 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405135+0000", 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405135+0000", 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405136+0000", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8299999999999995e-07 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405159+0000", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2906000000000001e-05 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405166+0000", 2024-03-04T16:05:58.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6159999999999996e-06 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405172+0000", 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8490000000000002e-06 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405239+0000", 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7040999999999997e-05 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.405260+0000", 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0808000000000001e-05 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:100004 7.e 7:7642d7fd:::benchmark_data_smithi038_77863_object100003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.358215+0000", 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.047461428999999999, 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047780841999999997, 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:05:58.449 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 100004 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358215+0000", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358215+0000", 2024-03-04T16:05:58.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358218+0000", 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.542e-06 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358245+0000", 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6689000000000001e-05 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:05:58.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358257+0000", 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1838000000000001e-05 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.358266+0000", 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.02e-06 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.365383+0000", 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071175759999999996 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.365765+0000", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038205999999999998 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.366291+0000", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052579600000000003 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.382103+0000", 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015811777999999999 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:05:58.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.382176+0000", 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3207000000000006e-05 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 5 2024-03-04T16:05:58.454 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:05:58.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:05:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.502 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:58.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:05:58.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:58.610 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:05:58 smithi038 ceph-mon[38113]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 785 op/s 2024-03-04T16:05:58.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:58.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:58.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:05:58 smithi176 ceph-mon[33619]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 785 op/s 2024-03-04T16:05:58.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:58.895 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:58.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:58.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:05:58 smithi038 ceph-mon[29779]: pgmap v118: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 785 op/s 2024-03-04T16:05:58.970 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 71 16 100551 100535 88.4472 54.375 0.0082233 0.0112977 2024-03-04T16:05:58.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:59.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:59.116 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:05:59.117 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:05:59.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:59.161 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:59.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:05:59.295 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:05:59.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:05:59.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:05:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.472 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:59.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:05:59.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:05:59.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:05:59.607 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:59.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:05:59.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:05:59.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:05:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:05:59.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:05:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:05:59.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:05:59.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:05:59.970 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 72 16 101370 101354 87.9296 51.1875 0.0141368 0.011366 2024-03-04T16:05:59.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:00.021 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:00.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:00.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:00.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:00.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:00.414 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:00.447 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:00.516 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:00.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:00.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:00.692 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:00.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:00.831 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:00.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:00 smithi038 ceph-mon[29779]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 975 op/s 2024-03-04T16:06:00.880 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:00 smithi038 ceph-mon[38113]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 975 op/s 2024-03-04T16:06:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:06:00.932 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:06:00.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-04T16:06:00.933 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:00.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:00.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:00.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:00.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 73 16 102256 102240 87.4829 55.375 0.00885422 0.011423 2024-03-04T16:06:01.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:01.089 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:00 smithi176 ceph-mon[33619]: pgmap v119: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 975 op/s 2024-03-04T16:06:01.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:01.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:01.374 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:01.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:01.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:01.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:01.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:01.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:01.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:01.785 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:01.873 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:01.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:01.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:01.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:01.971 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 74 16 103110 103094 87.0221 53.375 0.00587572 0.0114855 2024-03-04T16:06:02.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.068 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:02.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:02.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:02.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:02.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:02.307 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:02.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:02.410 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:02.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:02.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.486 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:02.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:02.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:02.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:02.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:02 smithi038 ceph-mon[29779]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-04T16:06:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:02 smithi038 ceph-mon[29779]: osdmap e118: 8 total, 8 up, 8 in 2024-03-04T16:06:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:02 smithi038 ceph-mon[38113]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-04T16:06:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:02 smithi038 ceph-mon[38113]: osdmap e118: 8 total, 8 up, 8 in 2024-03-04T16:06:02.760 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:02.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:02 smithi176 ceph-mon[33619]: pgmap v120: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s 2024-03-04T16:06:02.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:02 smithi176 ceph-mon[33619]: osdmap e118: 8 total, 8 up, 8 in 2024-03-04T16:06:02.861 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:02.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:02.887 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:02.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:02.972 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 75 15 103934 103919 86.5495 51.5625 0.0663594 0.0115463 2024-03-04T16:06:02.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:03.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:03.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:03.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:03.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:03.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:03.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:03.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:03.652 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:03.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:03.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:03.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:03.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:03.953 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:03.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:03.971 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 76 16 104845 104829 86.1592 56.875 0.0246685 0.0116015 2024-03-04T16:06:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:04.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:04.082 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:04.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.236 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:04.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:04.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:04.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:04.467 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:04.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:04.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:04.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:04.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:04.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:04.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:04 smithi038 ceph-mon[29779]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-04T16:06:04.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:04 smithi038 ceph-mon[38113]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-04T16:06:04.782 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:04.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:04.930 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:04.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:04.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:04.973 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 77 16 105650 105634 85.692 50.3125 0.00678472 0.0116623 2024-03-04T16:06:05.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:05.096 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:04 smithi176 ceph-mon[33619]: pgmap v122: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-04T16:06:05.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:05.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:05.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:05.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:05.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:05.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:05.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:05.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:05.697 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:06:05.718 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:05.749 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:05.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:05.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:05.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:05.973 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 78 16 106610 106594 85.3627 60 0.00828395 0.0117098 2024-03-04T16:06:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2390113013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:05 smithi038 ceph-mon[29779]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2390113013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:05 smithi038 ceph-mon[38113]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:05.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:06.027 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:06.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2390113013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:05 smithi176 ceph-mon[33619]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:06.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:06.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:06.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:06.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:06.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:06.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:06.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:06.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:06.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:06.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:06.721 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:06.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:06.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:06.943 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:06:06.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: pgmap v123: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 896 op/s 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: osdmap e119: 8 total, 8 up, 8 in 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: pgmap v123: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 896 op/s 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: osdmap e119: 8 total, 8 up, 8 in 2024-03-04T16:06:06.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:06 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:06.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:06.973 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 79 16 107628 107612 85.0874 63.625 0.0205682 0.0117471 2024-03-04T16:06:06.987 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:06:06.991 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:06.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:07.031 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:06:07.074 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:06:07.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: pgmap v123: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 56 MiB/s wr, 896 op/s 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: osdmap e119: 8 total, 8 up, 8 in 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:06 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:07.120 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:06:07.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:07.163 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:06:07.205 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:06:07.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:07.249 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:06:07.254 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:06:07.275 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:07.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:07.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:07.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:07.524 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:07.551 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:07.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:07.683 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:07.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.821 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:07.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:07.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:07.965 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:07.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:07.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:06:07.974361+0000 min lat: 0.00146693 max lat: 0.223515 avg lat: 0.0117826 2024-03-04T16:06:07.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:06:07.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 80 16 108670 108654 84.8379 65.125 0.00989315 0.0117826 2024-03-04T16:06:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:07 smithi038 ceph-mon[29779]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:06:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2134127667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:07 smithi038 ceph-mon[38113]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:06:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:07 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2134127667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:08.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.094 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:08.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:07 smithi176 ceph-mon[33619]: osdmap e120: 8 total, 8 up, 8 in 2024-03-04T16:06:08.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:07 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2134127667' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:08.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.235 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:08.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:08.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:08.364 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:08.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:08.503 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:08.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.641 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:08.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:08.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:08.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:08.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:08.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:08.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:08.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:08.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 81 16 109589 109573 84.4997 57.4375 0.00331673 0.0118253 2024-03-04T16:06:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:08 smithi038 ceph-mon[29779]: pgmap v126: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:06:08.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:08 smithi038 ceph-mon[38113]: pgmap v126: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:06:09.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:09.059 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:09.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:08 smithi176 ceph-mon[33619]: pgmap v126: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:06:09.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:09.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:09.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:09.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:09.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:09.465 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:09.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:09.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.602 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:09.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:09.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:09.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:09.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:09.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:09.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:09.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 82 16 110654 110638 84.281 66.5625 0.0113797 0.0118602 2024-03-04T16:06:09.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:10.026 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:10.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:10.180 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:10.282 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:10.313 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:10.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:10.441 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:10.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:10.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.591 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:10.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:10.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:10.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:10.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:10.765 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:10 smithi038 ceph-mon[29779]: pgmap v127: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-04T16:06:10.766 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:10 smithi038 ceph-mon[38113]: pgmap v127: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-04T16:06:10.777 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:06:10.777 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:06:10.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:10.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:10.966 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:10.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 83 16 111606 111590 83.9825 59.5 0.0126806 0.0119022 2024-03-04T16:06:10.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:10.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:10.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:11.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:10 smithi176 ceph-mon[33619]: pgmap v127: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-03-04T16:06:11.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:11.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:11.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:11.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:11.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:11.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:11.540 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:11.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:11.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.679 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:11.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:11.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:11.807 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:11.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:11.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:11.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 84 16 112490 112474 83.6405 55.25 0.0154384 0.0119509 2024-03-04T16:06:12.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:12.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:12.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.226 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:12.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:12.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:12.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:12.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:12.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:12.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:12.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:12.700 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:12.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:12.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:12.936 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:12.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:12.974 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 85 16 113452 113436 83.3639 60.125 0.00747521 0.0119909 2024-03-04T16:06:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:12 smithi038 ceph-mon[29779]: pgmap v128: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 571 op/s 2024-03-04T16:06:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:12 smithi038 ceph-mon[38113]: pgmap v128: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 571 op/s 2024-03-04T16:06:13.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:13.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:06:13.066 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:12 smithi176 ceph-mon[33619]: pgmap v128: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 36 MiB/s wr, 571 op/s 2024-03-04T16:06:13.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:13.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:13.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:13.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:13.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:13.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:13.517 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:13.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:13.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:06:13.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.648 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:13.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:13.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:13.776 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:13.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:13.938 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:13.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:13.975 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 86 16 114422 114406 83.0986 60.625 0.0151914 0.0120289 2024-03-04T16:06:13.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:14.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:14.068 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:14.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.217 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:14.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:14.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:14.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:14.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:14.497 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:14.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:14.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:14.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:14.756 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:14.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:14.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:14.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:14.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:14.975 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 87 16 115454 115438 82.8849 64.5 0.0212444 0.0120597 2024-03-04T16:06:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:14 smithi038 ceph-mon[29779]: pgmap v129: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-04T16:06:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:14 smithi038 ceph-mon[38113]: pgmap v129: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-04T16:06:15.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:15.036 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:14 smithi176 ceph-mon[33619]: pgmap v129: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.18k op/s 2024-03-04T16:06:15.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:15.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:15.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:15.299 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:15.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:15.438 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:15.541 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:15.542 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.568 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:15.669 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:15.696 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:15.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:15.826 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:15.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:15.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:15.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:15.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:15.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 88 16 116420 116404 82.6291 60.375 0.0197733 0.0120973 2024-03-04T16:06:16.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:16.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:06:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:16.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:16.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:16.249 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:16.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:16.366 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:06:16.366 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:06:16.367 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:06:16.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:16.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:16.591 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:16.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:16.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[29779]: pgmap v130: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 974 op/s 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[38113]: pgmap v130: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 974 op/s 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:16.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:16 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:16.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:16.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:16 smithi176 ceph-mon[33619]: pgmap v130: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 61 MiB/s wr, 974 op/s 2024-03-04T16:06:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:16 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:16 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:16 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:16 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:16.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:16.975 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 89 16 117331 117315 82.3405 56.9375 0.0160651 0.0121396 2024-03-04T16:06:16.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:16.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:17.100 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:17.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:17.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:17.259 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:17.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:17.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:17.498 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:17.541 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:17.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:17.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.679 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:17.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:17.728 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:06:17.728 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:06:17.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-04T16:06:17.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:17.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:17.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:17.934 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[29779]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[38113]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:17.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:17 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:17.956 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:17.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:17.975 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total time run: 90.0038 2024-03-04T16:06:17.975 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total writes made: 118265 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Write size: 65536 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object size: 65536 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Bandwidth (MB/sec): 82.125 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Bandwidth: 33.3258 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max bandwidth (MB/sec): 205.812 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min bandwidth (MB/sec): 40.6875 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average IOPS: 1314 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev IOPS: 533.212 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max IOPS: 3293 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min IOPS: 651 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average Latency(s): 0.0121725 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Latency(s): 0.0115917 2024-03-04T16:06:17.976 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max latency(s): 0.223515 2024-03-04T16:06:17.977 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min latency(s): 0.00146693 2024-03-04T16:06:17.983 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Cleaning up (deleting benchmark objects) 2024-03-04T16:06:18.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:18.092 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:17 smithi176 ceph-mon[33619]: osdmap e121: 8 total, 8 up, 8 in 2024-03-04T16:06:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:17 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:06:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:17 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:06:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:17 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:18.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:18.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:18.325 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:18.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:18.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:18.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:18.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:18.614 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:18.716 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:18.748 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:18.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:18.894 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:18.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:18.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:19.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:18 smithi038 ceph-mon[29779]: pgmap v132: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-04T16:06:19.071 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:18 smithi038 ceph-mon[38113]: pgmap v132: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-04T16:06:19.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:18 smithi176 ceph-mon[33619]: pgmap v132: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-04T16:06:19.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:19.156 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:19.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:19.288 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:19.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:19.418 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:19.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:19.551 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:19.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:19.681 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:19.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.808 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:19.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:19.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:19.951 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:19.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:20.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:20.103 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:20.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:20.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:20.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:20.380 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:20.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:20.513 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:20.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.656 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:20.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:20.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:20.785 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:20.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:20.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:20.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:21.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:21.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:21.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:20 smithi038 ceph-mon[29779]: pgmap v133: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:21.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:20 smithi038 ceph-mon[38113]: pgmap v133: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:20 smithi176 ceph-mon[33619]: pgmap v133: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:21.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:21.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:21.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:21.320 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:21.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:21.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:21.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:21.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:21.679 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:21.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:21.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:21.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:21.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:21.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:21.898 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:21 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:21.899 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:21 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:21.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:22.011 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:22.042 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:06:22.100 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:06:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:21 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:22.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:22.146 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:06:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.170 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:22.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:22.201 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:06:22.246 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:98561 7.e 7:793c9940:::benchmark_data_smithi038_77863_object98560:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:56.697596+0000", 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.548648085, 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18573724599999999, 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 98561 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697596+0000", 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697596+0000", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697599+0000", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.605e-06 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697622+0000", 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.298e-05 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.253 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697649+0000", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6889999999999998e-05 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.697659+0000", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0171e-05 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.739174+0000", 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.041515437000000002 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.254 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.739326+0000", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015197299999999999 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.739583+0000", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025629400000000001 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.743217+0000", 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036349049999999999 2024-03-04T16:06:22.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.743791+0000", 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00057314200000000001 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.883218+0000", 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13942792400000001 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.883274+0000", 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5578999999999998e-05 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:56.883333+0000", 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9345999999999997e-05 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:98979 7.e 7:74636674:::benchmark_data_smithi038_77863_object98978:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.257 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:57.175473+0000", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 25.070770897999999, 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.174656742, 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 98979 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175473+0000", 2024-03-04T16:06:22.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175473+0000", 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175476+0000", 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.26e-06 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175499+0000", 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3322000000000001e-05 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175515+0000", 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6422999999999998e-05 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.175524+0000", 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5750000000000001e-06 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.213410+0000", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037886073999999999 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.213517+0000", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010710899999999999 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.213664+0000", 2024-03-04T16:06:22.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014699099999999999 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.223783+0000", 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010119074 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.224112+0000", 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00032860499999999999 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.349977+0000", 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.12586557200000001 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.350051+0000", 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3343000000000006e-05 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.350130+0000", 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9394000000000006e-05 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:99564 7.e 7:7e062c2e:::benchmark_data_smithi038_77863_object99563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:57.821558+0000", 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.424686392000002, 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17994686600000001, 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 99564 2024-03-04T16:06:22.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821558+0000", 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821558+0000", 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821560+0000", 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.243e-06 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821585+0000", 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4664999999999998e-05 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821602+0000", 2024-03-04T16:06:22.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7258e-05 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.821611+0000", 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1900000000000001e-06 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.858544+0000", 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036932694000000002 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.858797+0000", 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025304199999999998 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.859297+0000", 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049976999999999997 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.869085+0000", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0097882109999999998 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.869129+0000", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4190000000000002e-05 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001372+0000", 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.13224279 2024-03-04T16:06:22.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001439+0000", 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7088999999999994e-05 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001505+0000", 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5723999999999996e-05 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:99577 7.e 7:78f2d35d:::benchmark_data_smithi038_77863_object99576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:57.838629+0000", 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 24.407615186000001, 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.163062755, 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 99577 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838629+0000", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838629+0000", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838631+0000", 2024-03-04T16:06:22.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8109999999999999e-06 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838655+0000", 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4258999999999999e-05 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838666+0000", 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.11e-05 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.838674+0000", 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5909999999999998e-06 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.864557+0000", 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025883269 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.864720+0000", 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016270400000000001 2024-03-04T16:06:22.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.864889+0000", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016937500000000001 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.869146+0000", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042565140000000003 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:57.869190+0000", 2024-03-04T16:06:22.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4113999999999998e-05 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001592+0000", 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.132401985 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001640+0000", 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8319999999999998e-05 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.001692+0000", 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1712999999999998e-05 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:100531 7.e 7:7cc513db:::benchmark_data_smithi038_77863_object100530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:58.943676+0000", 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 23.302567835000001, 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.158957825, 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.277 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 100531 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943676+0000", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943676+0000", 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943678+0000", 2024-03-04T16:06:22.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:22.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0379999999999998e-06 2024-03-04T16:06:22.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943701+0000", 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2804999999999999e-05 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943715+0000", 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3444999999999999e-05 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.943723+0000", 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7309999999999999e-06 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.967226+0000", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.023502927999999999 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.967385+0000", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015813399999999999 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.967550+0000", 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000165742 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.971952+0000", 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044017700000000002 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:58.972544+0000", 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059208799999999995 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.102473+0000", 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.129929192 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.102569+0000", 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5649000000000005e-05 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.102634+0000", 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5302999999999995e-05 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.284 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:100931 7.e 7:73c7bb8d:::benchmark_data_smithi038_77863_object100930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:59.396680+0000", 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.849563991, 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18504045599999999, 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 100931 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396680+0000", 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396680+0000", 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396682+0000", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2120000000000002e-06 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396705+0000", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2691999999999999e-05 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396721+0000", 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5852e-05 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.396736+0000", 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5088e-05 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.411013+0000", 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014276789999999999 2024-03-04T16:06:22.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.411425+0000", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000411612 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.412375+0000", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095010699999999995 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.416809+0000", 2024-03-04T16:06:22.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044347279999999998 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.417576+0000", 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076651400000000004 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.581589+0000", 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.164013626 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.581653+0000", 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3854000000000002e-05 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.581721+0000", 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7380999999999997e-05 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:101009 7.e 7:74bb8de0:::benchmark_data_smithi038_77863_object101008:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:59.485046+0000", 2024-03-04T16:06:22.291 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.761197761999998, 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16771143699999999, 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 101009 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485046+0000", 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485046+0000", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485049+0000", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8480000000000001e-06 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485076+0000", 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6826999999999999e-05 2024-03-04T16:06:22.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485089+0000", 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3055e-05 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.485098+0000", 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2280000000000004e-06 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.594574+0000", 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.109475656 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.594724+0000", 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000149705 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.594912+0000", 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000188102 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.610867+0000", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015955395000000001 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.610882+0000", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4619e-05 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652336+0000", 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.041454416000000001 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652697+0000", 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036071999999999998 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652758+0000", 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0866000000000003e-05 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:101011 7.e 7:799733aa:::benchmark_data_smithi038_77863_object101010:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:59.495035+0000", 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.751209538000001, 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15797530500000001, 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 101011 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495035+0000", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495035+0000", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495037+0000", 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.615e-06 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495062+0000", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4783e-05 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495081+0000", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8817e-05 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.495091+0000", 2024-03-04T16:06:22.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9690000000000006e-06 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.598426+0000", 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.103334906 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.598585+0000", 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000158999 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.598791+0000", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000206567 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.610917+0000", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012125761000000001 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.610929+0000", 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1851999999999999e-05 2024-03-04T16:06:22.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652855+0000", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.041926448999999998 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652952+0000", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6148000000000002e-05 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.653010+0000", 2024-03-04T16:06:22.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8439000000000001e-05 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:101084 7.e 7:76f14b37:::benchmark_data_smithi038_77863_object101083:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:59.593888+0000", 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.652356652000002, 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17072447099999999, 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.304 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 101084 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593888+0000", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593888+0000", 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593890+0000", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7690000000000001e-06 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593915+0000", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.446e-05 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593928+0000", 2024-03-04T16:06:22.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2811e-05 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.593936+0000", 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.704e-06 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.616195+0000", 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022258408 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.616662+0000", 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00046737299999999999 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.617112+0000", 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000449949 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.652817+0000", 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.035704519999999997 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.764489+0000", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.11167200300000001 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.764513+0000", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4331000000000001e-05 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.764560+0000", 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6801999999999997e-05 2024-03-04T16:06:22.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.764612+0000", 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2340999999999999e-05 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:101158 7.e 7:71a1be01:::benchmark_data_smithi038_77863_object101157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:59.688457+0000", 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 22.557787043000001, 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15791656200000001, 2024-03-04T16:06:22.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 101158 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688457+0000", 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688457+0000", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688458+0000", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9800000000000003e-07 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688477+0000", 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9347999999999998e-05 2024-03-04T16:06:22.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688487+0000", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6010000000000001e-06 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.688493+0000", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3629999999999999e-06 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.794392+0000", 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.105898564 2024-03-04T16:06:22.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.794529+0000", 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013759699999999999 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.794744+0000", 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000214081 2024-03-04T16:06:22.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.801829+0000", 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0070854689999999996 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.809256+0000", 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074273430000000003 2024-03-04T16:06:22.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.845970+0000", 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.036713889 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.846190+0000", 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000219402 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:59.846374+0000", 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000184107 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:102220 7.e 7:732bce0e:::benchmark_data_smithi038_77863_object102219:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:00.924755+0000", 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.321489749000001, 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15910781600000001, 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 102220 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924755+0000", 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924755+0000", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924757+0000", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5009999999999999e-06 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924799+0000", 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1839999999999999e-05 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924818+0000", 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8779000000000001e-05 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.924827+0000", 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6330000000000008e-06 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.953160+0000", 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.028332811999999999 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.953303+0000", 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142893 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.953483+0000", 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000180089 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.957871+0000", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043880309999999997 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:00.983296+0000", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.025424842 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083737+0000", 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.100441376 2024-03-04T16:06:22.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083794+0000", 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7108999999999998e-05 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083862+0000", 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7910999999999994e-05 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:102350 7.e 7:7a9685b3:::benchmark_data_smithi038_77863_object102349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:01.083144+0000", 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 21.163099765999998, 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.181762233, 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 102350 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083144+0000", 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083144+0000", 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083147+0000", 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1689999999999999e-06 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083172+0000", 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5253e-05 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083183+0000", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1100999999999999e-05 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.083191+0000", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.627e-06 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.114733+0000", 2024-03-04T16:06:22.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031542773000000003 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.114859+0000", 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012572700000000001 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.115016+0000", 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015714400000000001 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.118498+0000", 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034815029999999999 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.146024+0000", 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.027526388999999998 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.264738+0000", 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.118714007 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.264796+0000", 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7784000000000001e-05 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.264907+0000", 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011075599999999999 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:102806 7.e 7:7966e636:::benchmark_data_smithi038_77863_object102805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:01.637602+0000", 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.608642162999999, 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15557801800000001, 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 102806 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637602+0000", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637602+0000", 2024-03-04T16:06:22.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637604+0000", 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0499999999999999e-06 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637628+0000", 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4341e-05 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637639+0000", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0701e-05 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.637648+0000", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4549999999999995e-06 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.640814+0000", 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031666160000000001 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.640953+0000", 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013872099999999999 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.641117+0000", 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016380400000000001 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.649582+0000", 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0084652919999999993 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.662980+0000", 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013397632 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.792786+0000", 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.12980581599999999 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.792935+0000", 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014967499999999999 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.793180+0000", 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024491500000000001 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.338 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:102811 7.e 7:7dd9dc4e:::benchmark_data_smithi038_77863_object102810:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-04T16:06:22.339 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:06:22.339 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:01.639152+0000", 2024-03-04T16:06:22.339 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 20.607092482999999, 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15499204699999999, 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 102811 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639152+0000", 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639152+0000", 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639154+0000", 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0490000000000002e-06 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639177+0000", 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3424000000000001e-05 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639196+0000", 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8916999999999999e-05 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.639205+0000", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3510000000000002e-06 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.645001+0000", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057960679999999997 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.645144+0000", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014369 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.645312+0000", 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000167658 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.662745+0000", 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017432692 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.663020+0000", 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027529300000000002 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.793195+0000", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.130174755 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.793787+0000", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059276200000000004 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:01.794144+0000", 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00035638800000000002 2024-03-04T16:06:22.345 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:104354 7.e 7:777d737d:::benchmark_data_smithi038_77863_object104353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:03.444788+0000", 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.801455778000001, 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16509369900000001, 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 104354 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444788+0000", 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444788+0000", 2024-03-04T16:06:22.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444790+0000", 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5999999999999999e-06 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444813+0000", 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2518999999999999e-05 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444821+0000", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2630000000000003e-06 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.444827+0000", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.477e-06 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.534811+0000", 2024-03-04T16:06:22.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.089983460000000001 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.534933+0000", 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012260799999999999 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.535085+0000", 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000151743 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.584876+0000", 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.049791153999999997 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.584895+0000", 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8315e-05 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.609751+0000", 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024856729000000001 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.609825+0000", 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3430000000000007e-05 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.609882+0000", 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7401000000000002e-05 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.352 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:104383 7.e 7:76289cf9:::benchmark_data_smithi038_77863_object104382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:03.472003+0000", 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 18.774241571000001, 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.160073309, 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 104383 2024-03-04T16:06:22.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472003+0000", 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472003+0000", 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472005+0000", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0279999999999999e-06 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472040+0000", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4929999999999999e-05 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472055+0000", 2024-03-04T16:06:22.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5826999999999999e-05 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.472064+0000", 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8040000000000007e-06 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.538359+0000", 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.066295037000000001 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.538502+0000", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000142263 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.538676+0000", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017420700000000001 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.585039+0000", 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046363080000000001 2024-03-04T16:06:22.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.585054+0000", 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5203e-05 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.631851+0000", 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046796651000000002 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.632008+0000", 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015763800000000001 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:03.632076+0000", 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7640999999999998e-05 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:105468 7.e 7:74fcd721:::benchmark_data_smithi038_77863_object105467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.359 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:04.715437+0000", 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.530806870999999, 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.20431222099999999, 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 105468 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.360 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715437+0000", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715437+0000", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715440+0000", 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.176e-06 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.361 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715467+0000", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7569999999999999e-05 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715494+0000", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6486e-05 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.715503+0000", 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0769999999999997e-06 2024-03-04T16:06:22.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.731188+0000", 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015685372999999999 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.731321+0000", 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013307799999999999 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.731485+0000", 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000163974 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740299+0000", 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088138980000000006 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.777847+0000", 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.037548176000000003 2024-03-04T16:06:22.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.919631+0000", 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.14178428600000001 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.919686+0000", 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4518999999999998e-05 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.919750+0000", 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3608000000000005e-05 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:105487 7.e 7:7fe1e618:::benchmark_data_smithi038_77863_object105486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:04.740128+0000", 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.506116213999999, 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.18008358599999999, 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.366 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 105487 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740128+0000", 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740128+0000", 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740130+0000", 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.846e-06 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740158+0000", 2024-03-04T16:06:22.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8398000000000002e-05 2024-03-04T16:06:22.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740170+0000", 2024-03-04T16:06:22.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1851000000000001e-05 2024-03-04T16:06:22.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740178+0000", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5109999999999997e-06 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740332+0000", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015477700000000001 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740467+0000", 2024-03-04T16:06:22.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000134511 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.740618+0000", 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015125199999999999 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.745153+0000", 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045348530000000001 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.777882+0000", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.032728514 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.919803+0000", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.14192190099999999 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.920015+0000", 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021199599999999999 2024-03-04T16:06:22.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.920212+0000", 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000196176 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:105529 7.e 7:7193f8bd:::benchmark_data_smithi038_77863_object105528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:04.791843+0000", 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.454401643000001, 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16338180199999999, 2024-03-04T16:06:22.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 105529 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791843+0000", 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791843+0000", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791844+0000", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7379999999999999e-06 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791880+0000", 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5936999999999998e-05 2024-03-04T16:06:22.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791891+0000", 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0274e-05 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.791899+0000", 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3580000000000003e-06 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.925384+0000", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.133485086 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.928926+0000", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035420909999999998 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.929105+0000", 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000178526 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.937437+0000", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083323960000000006 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.937492+0000", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4625e-05 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.954828+0000", 2024-03-04T16:06:22.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017336313999999999 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.955035+0000", 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020731600000000001 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.955224+0000", 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018914100000000001 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:105540 7.e 7:7cc379b7:::benchmark_data_smithi038_77863_object105539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:04.799263+0000", 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 17.446981739999998, 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.156873543, 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 105540 2024-03-04T16:06:22.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799263+0000", 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799263+0000", 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799264+0000", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.612e-06 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799288+0000", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3952e-05 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799301+0000", 2024-03-04T16:06:22.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2709e-05 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.799308+0000", 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0269999999999998e-06 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.933215+0000", 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.133907312 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.933398+0000", 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00018260800000000001 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.933568+0000", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017021199999999999 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.937644+0000", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040764920000000001 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.954667+0000", 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017022745999999998 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.955686+0000", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010187919999999999 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.955883+0000", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019658800000000001 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:04.956136+0000", 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025349299999999998 2024-03-04T16:06:22.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:06:22.391 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:06:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.392 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:22.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:06:22.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:06:22.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:22.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:22.623 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:22.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:22.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:22.778 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:22.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:22.812 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:06:22.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:22.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:22.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[29779]: pgmap v134: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:06:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1699981139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[38113]: pgmap v134: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:22.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:06:22.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1699981139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:23.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.068 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:23.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:22 smithi176 ceph-mon[33619]: pgmap v134: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-04T16:06:23.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:06:23.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1699981139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:23.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.220 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:23.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:23.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:23.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:23.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:23.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:23.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:23.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.625 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:23.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:23.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:23.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:23.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: osdmap e122: 8 total, 8 up, 8 in 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:06:23.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: osdmap e122: 8 total, 8 up, 8 in 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:06:23.891 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:23 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:06:23.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:23.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:23.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:24.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.040 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:24.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/367879854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: osdmap e122: 8 total, 8 up, 8 in 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:06:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:06:24.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:06:24.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:23 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:06:24.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:24.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:24.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:24.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:06:24.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:24.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:24.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:24.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:24.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:24.554 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:24.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:24.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.737 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:24.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:24.839 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:24.870 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:24.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:24.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:06:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:06:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: pgmap v136: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-03-04T16:06:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:06:24.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: pgmap v136: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:06:24.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:24 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:06:25.003 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:25.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: pgmap v136: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.26k op/s 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:06:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:24 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:06:25.135 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:25.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:25.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:25.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:25.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:25.394 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:25.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.536 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:25.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:25.637 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:25.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:25.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:25.811 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:25.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:25.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:25.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:26.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.092 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:26.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:26.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:26.216 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:26.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:26.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:26.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:26.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:26.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:26.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:06:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:06:26.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:06:26.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:06:26.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:26 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:06:26.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:06:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:26 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:06:26.624 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:26.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:26.767 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:26.868 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:26.898 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:26.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:27.000 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:27.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:27.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:27.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: pgmap v137: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 1.18k op/s 2024-03-04T16:06:27.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: 7.e scrub starts 2024-03-04T16:06:27.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:06:27.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:06:27.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: pgmap v137: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 1.18k op/s 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: 7.e scrub starts 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:06:27.175 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:27 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:06:27.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:27.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:27.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:06:27.452 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:06:27.454 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:27.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.516 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:27.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: pgmap v137: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 39 MiB/s wr, 1.18k op/s 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: 7.e scrub starts 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:06:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:27 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:06:27.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:27.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:27.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:27.776 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:27.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:27.908 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:27.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:28.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.041 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:28.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:28.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:28.179 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:28.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:28.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:06:28.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:06:28.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:06:28.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:06:28.351 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:06:28.351 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:06:28.351 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:06:28.351 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:28 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:06:28.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.490 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:28.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:28.512 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:28 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:06:28.512 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:28 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:06:28.512 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:28 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:06:28.512 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:28 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:06:28.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:28.618 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:28.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:06:28.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:28.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:28.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:06:28.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:28.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:28.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:28.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:29.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:29.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:29.160 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:06:29.161 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:06:29.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-04T16:06:29.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:29.260 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:29.337 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: pgmap v138: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 1.03k op/s 2024-03-04T16:06:29.337 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:06:29.337 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[29779]: 7.e scrub ok 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: pgmap v138: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 1.03k op/s 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:06:29.338 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:29 smithi038 ceph-mon[38113]: 7.e scrub ok 2024-03-04T16:06:29.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:29.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:29.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:29.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.524 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:29.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: pgmap v138: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 22 MiB/s wr, 1.03k op/s 2024-03-04T16:06:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:06:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:06:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:06:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:06:29.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:29 smithi176 ceph-mon[33619]: 7.e scrub ok 2024-03-04T16:06:29.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:29.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:29.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:29.755 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:29.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:29.764 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:29.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:29.821 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:29.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:29.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:29.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:30.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:30.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:30.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:30.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:30.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:30.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:30.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[38113]: 7.0 scrub starts 2024-03-04T16:06:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[38113]: 7.4 scrub starts 2024-03-04T16:06:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[38113]: 7.7 scrub starts 2024-03-04T16:06:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[29779]: 7.0 scrub starts 2024-03-04T16:06:30.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[29779]: 7.4 scrub starts 2024-03-04T16:06:30.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:30 smithi038 ceph-mon[29779]: 7.7 scrub starts 2024-03-04T16:06:30.510 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:30 smithi176 ceph-mon[33619]: 7.0 scrub starts 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:30 smithi176 ceph-mon[33619]: 7.4 scrub starts 2024-03-04T16:06:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:30 smithi176 ceph-mon[33619]: 7.7 scrub starts 2024-03-04T16:06:30.612 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:30.639 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:30.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:30.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:30.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:30.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:30.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:31.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.068 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:31.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:31.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:31.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:31.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:31 smithi038 ceph-mon[29779]: pgmap v139: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:31.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:31 smithi038 ceph-mon[29779]: 7.0 scrub ok 2024-03-04T16:06:31.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:31 smithi038 ceph-mon[38113]: pgmap v139: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:31.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:31 smithi038 ceph-mon[38113]: 7.0 scrub ok 2024-03-04T16:06:31.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:31.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:31.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:31.463 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:31.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:31.595 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:31 smithi176 ceph-mon[33619]: pgmap v139: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:31 smithi176 ceph-mon[33619]: 7.0 scrub ok 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132206 7.7 e122/95 7:e0b4a543:::benchmark_data_smithi038_77863_object13938:head v 122'8164, mlcod=122'8162)", 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.671040+0000", 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.010204302, 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010235599999999999, 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671040+0000", 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671040+0000", 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671041+0000", 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3659999999999999e-06 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671049+0000", 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9070000000000007e-06 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671099+0000", 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0099000000000003e-05 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671112+0000", 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2322e-05 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671244+0000", 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013263500000000001 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.671263+0000", 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9088000000000001e-05 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132209 7.c e122/95 7:3a3c9b7a:::benchmark_data_smithi038_77863_object13941:head v 122'8180, mlcod=122'8178)", 2024-03-04T16:06:31.687 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.672511+0000", 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0087330570000000007, 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0088332559999999994, 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672511+0000", 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672511+0000", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672512+0000", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1620000000000001e-06 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672519+0000", 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8229999999999997e-06 2024-03-04T16:06:31.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672531+0000", 2024-03-04T16:06:31.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1695000000000001e-05 2024-03-04T16:06:31.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672537+0000", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3539999999999996e-06 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672598+0000", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0294999999999998e-05 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.672615+0000", 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7929999999999999e-05 2024-03-04T16:06:31.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132205 7.f e122/95 7:f09aa764:::benchmark_data_smithi038_77863_object13937:head v 122'8233, mlcod=122'8229)", 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.676351+0000", 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.004893059, 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050580740000000001, 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676351+0000", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676351+0000", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676353+0000", 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4470000000000001e-06 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676361+0000", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4090000000000003e-06 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676376+0000", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4474000000000001e-05 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676383+0000", 2024-03-04T16:06:31.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4089999999999997e-06 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676424+0000", 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1174999999999998e-05 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676443+0000", 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8969000000000001e-05 2024-03-04T16:06:31.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132208 7.f e122/95)", 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.676397+0000", 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0048470400000000004, 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050940239999999999, 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:06:31.696 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676397+0000", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676397+0000", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676398+0000", 2024-03-04T16:06:31.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6700000000000003e-07 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676402+0000", 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6670000000000004e-06 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676409+0000", 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4250000000000003e-06 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676415+0000", 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6899999999999997e-06 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:31.700 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:132211 7.3 7:c8937732:::benchmark_data_smithi038_77863_object13943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-04T16:06:31.700 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.676838+0000", 2024-03-04T16:06:31.700 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0044067100000000003, 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0047291360000000001, 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 132211 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676838+0000", 2024-03-04T16:06:31.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676838+0000", 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676840+0000", 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1679999999999998e-06 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676845+0000", 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5210000000000002e-06 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676856+0000", 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1148000000000001e-05 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676864+0000", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9440000000000005e-06 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.676990+0000", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012578599999999999 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677105+0000", 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114721 2024-03-04T16:06:31.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2,4", 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677219+0000", 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011405599999999999 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132212 7.1 e122/95 7:81819c6a:::benchmark_data_smithi038_77863_object13944:head v 122'8204, mlcod=122'8202)", 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.677358+0000", 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0038864429999999998, 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004296406, 2024-03-04T16:06:31.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677358+0000", 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677358+0000", 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677359+0000", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.232e-06 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677366+0000", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8430000000000004e-06 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677374+0000", 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7379999999999994e-06 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677380+0000", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6760000000000001e-06 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677430+0000", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9811000000000001e-05 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.677518+0000", 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7872000000000002e-05 2024-03-04T16:06:31.708 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132215 7.1 e122/95 7:8be3d13b:::benchmark_data_smithi038_77863_object13947:head v 122'8205, mlcod=122'8202)", 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.680122+0000", 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0011218689999999999, 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016053090000000001, 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680122+0000", 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680122+0000", 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680124+0000", 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1570000000000001e-06 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680131+0000", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.956e-06 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680140+0000", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7839999999999998e-06 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680147+0000", 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8360000000000003e-06 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680756+0000", 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060877000000000001 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680774+0000", 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7578e-05 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.712 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.14844.0:132211 7.3 e122/95)", 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.680166+0000", 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0010781429999999999, 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001627437, 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680166+0000", 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.713 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680166+0000", 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680167+0000", 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.27e-07 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680169+0000", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.886e-06 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680175+0000", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3580000000000002e-06 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680180+0000", 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7500000000000003e-06 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.715 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.14844.0:132214 7.7 e122/95 7:eb8392ee:::benchmark_data_smithi038_77863_object13946:head v 122'8165, mlcod=122'8162)", 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:06:31.680198+0000", 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0010464389999999999, 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001654612, 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:06:31.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680198+0000", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680198+0000", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680198+0000", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9999999999999997e-07 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:06:31.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680203+0000", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5870000000000003e-06 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680209+0000", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0519999999999998e-06 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680214+0000", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6820000000000002e-06 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:06:31.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680270+0000", 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5819999999999997e-05 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:06:31.680292+0000", 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2235999999999999e-05 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 9 2024-03-04T16:06:31.719 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:06:31.749 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:31.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:31.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:31.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:31.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:31.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:32.025 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:32.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:32.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:32.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.305 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:32.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:32.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:32.438 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:32.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 7.c scrub starts 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 7.7 scrub ok 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:06:32.467 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-mon[33619]: 7.4 scrub ok 2024-03-04T16:06:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 7.c scrub starts 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 7.7 scrub ok 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[29779]: 7.4 scrub ok 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:06:32.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 7.c scrub starts 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 7.7 scrub ok 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: pgmap v140: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 3.1 MiB/s wr, 1.05k op/s 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:06:32.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:32 smithi038 ceph-mon[38113]: 7.4 scrub ok 2024-03-04T16:06:32.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:32.580 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:32.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:32.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:06:32.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:32.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.710 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:32.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:32.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:32.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:32.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:32.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:32.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:33.034 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:33.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.183 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:33.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:33.285 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:33.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:33.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:33.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:33.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:33.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[29779]: 7.c scrub ok 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:06:33.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:06:33.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:06:33.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:33.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:33 smithi038 ceph-mon[38113]: 7.c scrub ok 2024-03-04T16:06:33.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:33.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:06:33.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:33.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:33 smithi176 ceph-mon[33619]: 7.c scrub ok 2024-03-04T16:06:33.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:33.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:33.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:33.871 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:33.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:33.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:34.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:34.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:34.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.168 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:34.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:34.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:34.304 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:34.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:34.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:34.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:34 smithi038 ceph-mon[29779]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 971 B/s rd, 0 B/s wr, 954 op/s 2024-03-04T16:06:34.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:34 smithi038 ceph-mon[29779]: 7.6 scrub starts 2024-03-04T16:06:34.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:34 smithi038 ceph-mon[38113]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 971 B/s rd, 0 B/s wr, 954 op/s 2024-03-04T16:06:34.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:34 smithi038 ceph-mon[38113]: 7.6 scrub starts 2024-03-04T16:06:34.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.565 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:34.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:34 smithi176 ceph-mon[33619]: pgmap v141: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 971 B/s rd, 0 B/s wr, 954 op/s 2024-03-04T16:06:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:34 smithi176 ceph-mon[33619]: 7.6 scrub starts 2024-03-04T16:06:34.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.697 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:34.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:34.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:34.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:34.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:34.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:34.972 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:34.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:35.074 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:35.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:35.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:35.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:35 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:06:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:35 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:06:35.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:35 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:06:35.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:35 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:06:35.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:35.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:35.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:35.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:35 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:06:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:35 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:06:35.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.656 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:35.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:35.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:35.787 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:35.889 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:35.919 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:35.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:36.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:36.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:36.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:36.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:36.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:36.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.340 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:36.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:36.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.472 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:36.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:36.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[29779]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:06:36.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:06:36.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:06:36.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[38113]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:06:36.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:06:36.482 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:36 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:06:36.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:36 smithi176 ceph-mon[33619]: pgmap v142: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:06:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:36 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:06:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:36 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:06:36.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:36.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:36.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:36.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:36.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:36.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:36.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:36.991 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:37.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:37.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:37.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:37.266 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:37.304 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:37.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:37.437 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:37.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:37.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.563 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:37.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:06:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:06:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:06:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:06:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:06:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:06:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 7.6 scrub ok 2024-03-04T16:06:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:37 smithi176 ceph-mon[33619]: 7.f scrub starts 2024-03-04T16:06:37.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:37.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:06:37.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 7.6 scrub ok 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[29779]: 7.f scrub starts 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 7.6 scrub ok 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:37.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:37 smithi038 ceph-mon[38113]: 7.f scrub starts 2024-03-04T16:06:37.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:37.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:37.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:37.972 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:37.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:38.074 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:38.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:38.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:38.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:38.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:38.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:38.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:38.515 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:38.528 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:38 smithi176 ceph-mon[33619]: pgmap v143: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:38.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:38.646 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:38 smithi038 ceph-mon[29779]: pgmap v143: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:38 smithi038 ceph-mon[38113]: pgmap v143: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:38.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:38.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:38.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:38.926 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:38.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:39.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:39.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:39.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:39.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:06:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:06:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:06:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:06:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:39 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:06:39.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:39.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:39.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.461 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:39.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:39.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:39.602 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:06:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:39 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:06:39.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:39.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:39.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:39.880 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:39.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:39.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:40.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:40.162 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:40.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:40.307 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:40.308 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:06:40.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:40.437 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:40.467 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:06:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1311858496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[29779]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1311858496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:40 smithi038 ceph-mon[38113]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:06:40.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:40.554 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:06:40.577 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:40.603 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: pgmap v144: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1311858496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:40 smithi176 ceph-mon[33619]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:06:40.645 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:06:40.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:40.692 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:06:40.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:40.736 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:06:40.790 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:06:40.800 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:06:40.801 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:06:40.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:40.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:40.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:40.960 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:06:40.960 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:06:40.960 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:06:40.961 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:06:40.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:41.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:41.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:41.132 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:41.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:41.263 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:41.364 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:41.405 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:41.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.552 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:41.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:06:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:06:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: 7.f scrub ok 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: osdmap e123: 8 total, 8 up, 8 in 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2319312520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:41.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:41 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: 7.f scrub ok 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: osdmap e123: 8 total, 8 up, 8 in 2024-03-04T16:06:41.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2319312520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: 7.f scrub ok 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: osdmap e123: 8 total, 8 up, 8 in 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2319312520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:41.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:41 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:06:41.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:41.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:41.801 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.832 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:41.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:41.933 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:41.965 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:41.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:42.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:42.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:42.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:42.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:42.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:42.387 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:42.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: osdmap e124: 8 total, 8 up, 8 in 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:06:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[29779]: 7.9 scrub starts 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: osdmap e124: 8 total, 8 up, 8 in 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-04T16:06:42.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:06:42.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:06:42.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:42 smithi038 ceph-mon[38113]: 7.9 scrub starts 2024-03-04T16:06:42.530 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:42.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: osdmap e124: 8 total, 8 up, 8 in 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 637 op/s 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:06:42.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:42 smithi176 ceph-mon[33619]: 7.9 scrub starts 2024-03-04T16:06:42.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:42.660 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:42.762 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:42.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:42.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:42.925 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:42.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:43.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.072 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:43.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:43.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:43.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:43.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:43.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:43.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:43.463 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:43.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:06:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:06:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.1f deep-scrub starts 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.1f deep-scrub ok 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 7.b deep-scrub starts 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:06:43.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:43 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:06:43.611 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.1f deep-scrub starts 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.1f deep-scrub ok 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:06:43.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 7.b deep-scrub starts 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:06:43.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.1f deep-scrub starts 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.1f deep-scrub ok 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 7.b deep-scrub starts 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:06:43.700 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:43.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:43.742 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:43.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:43.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:06:43.828 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:06:43.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-04T16:06:43.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:43.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:43.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:43.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:44.002 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:44.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.151 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:44.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:44.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:06:44.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.286 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:44.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:44.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.431 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:44.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[29779]: 7.9 scrub ok 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:06:44.530 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:06:44.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:06:44.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:06:44.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-04T16:06:44.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:44 smithi038 ceph-mon[38113]: 7.9 scrub ok 2024-03-04T16:06:44.531 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:06:44.531 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:06:44.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:44.554 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:44.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:44.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:44.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:44.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:44.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:06:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:06:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:06:44.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:06:44.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-04T16:06:44.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:44 smithi176 ceph-mon[33619]: 7.9 scrub ok 2024-03-04T16:06:44.864 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:44.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:06:44.966 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:44.996 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:44.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:45.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:45.126 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:45.227 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:45.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.284 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:45.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:45.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:45.417 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 7.5 scrub starts 2024-03-04T16:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:06:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:06:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:06:45.500 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 7.5 scrub starts 2024-03-04T16:06:45.500 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:06:45.500 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:06:45.501 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:06:45.501 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:06:45.501 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:06:45.501 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:45 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:06:45.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:45.552 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:45.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:45.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:06:45.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:45.739 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:45.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 7.5 scrub starts 2024-03-04T16:06:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:06:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:06:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:06:45.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:06:45.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:06:45.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:45 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:06:45.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:45.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:45.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:45.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:46.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:46.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:46.152 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:46.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:46.297 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:46.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:46.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:46.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.571 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:46.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:46.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:46.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 3.c deep-scrub starts 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 3.c deep-scrub ok 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s 2024-03-04T16:06:46.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 3.c deep-scrub starts 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 3.c deep-scrub ok 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:06:46.730 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:46 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:06:46.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:46.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:46.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 3.c deep-scrub starts 2024-03-04T16:06:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 3.c deep-scrub ok 2024-03-04T16:06:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:06:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:06:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 992 op/s 2024-03-04T16:06:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:06:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:46 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:06:46.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:46.989 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:46.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:46.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:47.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.131 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:47.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:47.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:47.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:47.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:47.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:47.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:47.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:47.523 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:47.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:47.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 7.5 scrub ok 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:06:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 6.9 deep-scrub starts 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[29779]: 6.9 deep-scrub ok 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 7.5 scrub ok 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 6.9 deep-scrub starts 2024-03-04T16:06:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:47 smithi038 ceph-mon[38113]: 6.9 deep-scrub ok 2024-03-04T16:06:47.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:47.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 7.5 scrub ok 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:06:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 6.9 deep-scrub starts 2024-03-04T16:06:47.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:47 smithi176 ceph-mon[33619]: 6.9 deep-scrub ok 2024-03-04T16:06:47.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:47.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:47.938 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:47.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:48.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:48.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:48.186 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:48.220 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:48.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.357 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:48.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:48.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.490 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:48.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:48.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:48.608 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:48.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:48.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:48.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[29779]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 798 op/s 2024-03-04T16:06:48.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:06:48.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:06:48.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[38113]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 798 op/s 2024-03-04T16:06:48.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:06:48.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:48 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:06:48.765 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:48 smithi176 ceph-mon[33619]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 798 op/s 2024-03-04T16:06:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:48 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:06:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:48 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:06:48.868 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:48.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:48.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:48.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:49.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:49.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:49.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:49.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.310 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:49.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:49.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:49.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:49.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:49.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:49.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:06:49.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:06:49.609 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:49 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:06:49.635 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:49.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:49.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:49.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:49 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:06:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:49 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:06:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:49 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:06:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:49 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:06:49.867 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:49.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:49.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:50.013 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:50.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:50.144 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:50.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:50.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:50.272 INFO:teuthology.orchestra.run.smithi038.stderr:no valid command found; 10 closest matches: 2024-03-04T16:06:50.272 INFO:teuthology.orchestra.run.smithi038.stderr:0 2024-03-04T16:06:50.272 INFO:teuthology.orchestra.run.smithi038.stderr:1 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:2 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:abort 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:assert 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:bench [] [] [] [] 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs debug_inject_read_zeros 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs files list 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:bluefs stats 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:bluestore allocator dump block 2024-03-04T16:06:50.273 INFO:teuthology.orchestra.run.smithi038.stderr:Error EINVAL: invalid command 2024-03-04T16:06:50.280 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:50.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:50.411 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 3.7 deep-scrub starts 2024-03-04T16:06:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 3.7 deep-scrub ok 2024-03-04T16:06:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 6.3 deep-scrub starts 2024-03-04T16:06:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 6.3 deep-scrub ok 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 3.7 deep-scrub starts 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 3.7 deep-scrub ok 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 6.3 deep-scrub starts 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 6.3 deep-scrub ok 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:06:50.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:50 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:06:50.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:50.547 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:50.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:50.678 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:50.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:50.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 3.7 deep-scrub starts 2024-03-04T16:06:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 3.7 deep-scrub ok 2024-03-04T16:06:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:06:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 6.3 deep-scrub starts 2024-03-04T16:06:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 6.3 deep-scrub ok 2024-03-04T16:06:50.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:06:50.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:50 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:06:50.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:50.947 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:50.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:51.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.081 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:51.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:51.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:51.223 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:51.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:51.362 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:51.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:51.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:51.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:51.631 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:51.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[29779]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[38113]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:06:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:51 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:06:51.782 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:51 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:06:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:51 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:06:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:51 smithi176 ceph-mon[33619]: osdmap e125: 8 total, 8 up, 8 in 2024-03-04T16:06:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:51 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:06:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:51 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:06:51.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:51.913 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:51.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-04T16:06:51.993 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-04T16:06:52.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.052 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:52.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:52.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:52.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:52.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:52.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:52.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:52.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:52.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:52.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:52.678 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:06:52.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.743 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:52.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:52.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:06:52.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:06:52.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:06:52.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4260667466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[29779]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:52.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4260667466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:52 smithi038 ceph-mon[38113]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:52.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:06:52.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:06:52.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 908 op/s 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:06:52.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4260667466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:52 smithi176 ceph-mon[33619]: from='client.34853 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:06:52.873 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:52.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:52.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:52.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:53.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:53.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:53.140 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:53.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:53.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:53.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:53.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:06:53.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:06:53.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:06:53.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:06:53.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:53.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:53.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:06:53.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[29779]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:06:53.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:06:53.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:06:53.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:06:53.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:06:53.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:53.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:53.497 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:06:53.498 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-mon[38113]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:06:53.498 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:06:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:06:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:06:53.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:53.547 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:53.649 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:53.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:53.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:53.810 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:06:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:06:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:06:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:06:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:06:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:06:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: from='client.34853 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:06:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:53 smithi176 ceph-mon[33619]: osdmap e126: 8 total, 8 up, 8 in 2024-03-04T16:06:53.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:53.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:53.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:54.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.098 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:54.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:54.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:54.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:54.387 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:54.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:54.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:54.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:54.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:54.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:54.874 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:54 smithi038 ceph-mon[29779]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-04T16:06:54.875 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:54 smithi038 ceph-mon[38113]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-04T16:06:54.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:54.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:54.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:54.979 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:06:54.979 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:06:55.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:54 smithi176 ceph-mon[33619]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-04T16:06:55.181 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:55.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:55.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:55.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:55.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:55.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:55.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:55.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:55.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:06:55.609 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:55.660 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:06:55.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:55.713 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:06:55.739 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:55.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:55.758 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:06:55.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:06:55.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:55.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:06:55.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:06:55.894 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:55.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:55.929 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:06:55.968 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:06:55.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:06:55.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:06:55.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:06:55.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:06:55.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:06:55.979 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:06:55.979 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:06:55.979 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:06:55.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:56.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:56.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:06:56.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:56.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:56.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:56 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2459642955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:56.226 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:56 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2459642955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:56.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.298 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:56.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:56 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2459642955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:06:56.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:56.412 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:06:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:56.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:56.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.580 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:56.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:56.710 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:56.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:56.840 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:56.886 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:56.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:06:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:56.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:56.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:06:56.994 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:06:56.995 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:06:56.995 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.2569816671582034 2024-03-04T16:06:56.995 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 0.2569816671582034 2024-03-04T16:06:57.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:57.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:57.230 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:06:57.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:06:57.321 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:57 smithi038 ceph-mon[29779]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 936 op/s 2024-03-04T16:06:57.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:57 smithi038 ceph-mon[38113]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 936 op/s 2024-03-04T16:06:57.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:57.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:57.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:57.546 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:57.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:57 smithi176 ceph-mon[33619]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 936 op/s 2024-03-04T16:06:57.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:57.685 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:57.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:57.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:57.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:57.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:57.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:58.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.102 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:58.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:58.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:58.208 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:06:58.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:58.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:58.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:58.396 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:06:58.396 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:06:58.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:157690 7.1 7:87a24226:::benchmark_data_smithi038_77863_object39422:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:58.387801+0000", 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0078884479999999993, 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079224859999999994, 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 157690 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:06:58.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387801+0000", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387801+0000", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387803+0000", 2024-03-04T16:06:58.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3429999999999998e-06 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387811+0000", 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4070000000000004e-06 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387828+0000", 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7059999999999999e-05 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:06:58.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387838+0000", 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0332e-05 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.387905+0000", 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6890999999999994e-05 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.388036+0000", 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013104999999999999 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.388155+0000", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119539 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391561+0000", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003405726 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.395659+0000", 2024-03-04T16:06:58.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040980540000000003 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.395707+0000", 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8109999999999998e-05 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:157692 7.1 7:8cdda46f:::benchmark_data_smithi038_77863_object39424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:58.391878+0000", 2024-03-04T16:06:58.402 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.003811231, 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039472140000000001, 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 157692 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391878+0000", 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391878+0000", 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391880+0000", 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3199999999999998e-06 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:06:58.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391892+0000", 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1569999999999999e-05 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391911+0000", 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8678e-05 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:06:58.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391921+0000", 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0596e-05 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.391978+0000", 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7318000000000002e-05 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.392089+0000", 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011018000000000001 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.392214+0000", 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000125666 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop_reply(client.14844.0:157690 7.1 e126/95 ondisk, result = 0)", 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:58.394103+0000", 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.001586105, 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001795161, 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:06:58.407 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394103+0000", 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394103+0000", 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:06:58.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394104+0000", 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3799999999999999e-06 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394110+0000", 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.096e-06 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394123+0000", 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3658999999999999e-05 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.394131+0000", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.447e-06 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.395697+0000", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015660470000000001 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:58.395703+0000", 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8390000000000003e-06 2024-03-04T16:06:58.410 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 3 2024-03-04T16:06:58.411 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:06:58.421 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:58.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:58.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:06:58.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:06:58.728 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:06:58.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:58.875 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:58.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:58.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:59.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:59.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:06:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.170 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:59.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:06:59.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:06:59.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:06:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:59.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:06:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:06:59 smithi176 ceph-mon[33619]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 824 op/s 2024-03-04T16:06:59.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:06:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.466 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:59.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:06:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:06:59 smithi038 ceph-mon[29779]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 824 op/s 2024-03-04T16:06:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:06:59 smithi038 ceph-mon[38113]: pgmap v157: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 824 op/s 2024-03-04T16:06:59.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:59.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:59.600 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:06:59.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:06:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.740 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:06:59.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:06:59.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:06:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:06:59.878 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:06:59.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:06:59.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:00.015 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:00.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:00.154 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:00.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:00.293 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:00.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.418 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:00.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:00.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.552 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:00.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:00.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:00.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:00.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:00.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:00.945 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:00.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:01.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.087 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:01.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:01.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:01.219 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:01.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:01 smithi176 ceph-mon[33619]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:07:01.365 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:01.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:01 smithi038 ceph-mon[29779]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:07:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:01 smithi038 ceph-mon[38113]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:07:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:01.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:01.638 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:01.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:01.773 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:01.826 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:01.826 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:07:01.826 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:07:01.826 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:07:01.826 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:01.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:01.904 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:01.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:02.005 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:02.038 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:02.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:02.184 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:02.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:02.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.321 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:02.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:02.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:02.451 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:02.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:07:02.552 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:02.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:02.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:02.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:02.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:02.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:02.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:02.872 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:02.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:02.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:03.013 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:03.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:03.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:03.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:03.278 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:03 smithi176 ceph-mon[33619]: pgmap v159: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 957 op/s 2024-03-04T16:07:03.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.327 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:03.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:03.428 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:03.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:03.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:03.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:03 smithi038 ceph-mon[38113]: pgmap v159: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 957 op/s 2024-03-04T16:07:03.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:03 smithi038 ceph-mon[29779]: pgmap v159: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 957 op/s 2024-03-04T16:07:03.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:03.607 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:03.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:03.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:03.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:03.884 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:03.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:03.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.043 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:04.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:04.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.176 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:04.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:04.238 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:04.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:04.305 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:04.390 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:04.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:04.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:04.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:04.558 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:04.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:04.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.739 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:04.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:04.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:04.876 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:04.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:04.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:04.978 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.019 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:05.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.160 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:05.184 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:07:05.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:8980 7.1 7:85485f01:::benchmark_data_smithi038_77863_object8979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.17436223199999, 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075430288999999998, 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 8980 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.185 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004853+0000", 2024-03-04T16:07:05.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1570000000000001e-06 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004884+0000", 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0892999999999997e-05 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004896+0000", 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1905999999999999e-05 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004903+0000", 2024-03-04T16:07:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4950000000000002e-06 2024-03-04T16:07:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004952+0000", 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8758999999999999e-05 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005044+0000", 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2423999999999995e-05 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005176+0000", 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131514 2024-03-04T16:07:05.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005795+0000", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061942300000000002 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.006112+0000", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031637899999999999 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080171+0000", 2024-03-04T16:07:05.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074059199000000006 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080236+0000", 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5553000000000001e-05 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080282+0000", 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5587000000000002e-05 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.191 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23564 7.1 7:8f2d6c93:::benchmark_data_smithi038_77863_object23563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.96194910200001, 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067845241000000001, 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.192 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23564 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217273+0000", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7649999999999993e-06 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217320+0000", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7490000000000001e-05 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217331+0000", 2024-03-04T16:07:05.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0560000000000001e-05 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217337+0000", 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8399999999999997e-06 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217396+0000", 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8226000000000001e-05 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217522+0000", 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012583599999999999 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217665+0000", 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014378200000000001 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.219333+0000", 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001667647 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.219511+0000", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017759899999999999 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285008+0000", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065497256000000004 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285064+0000", 2024-03-04T16:07:05.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6240999999999999e-05 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285110+0000", 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5998999999999998e-05 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23579 7.1 7:8c29c604:::benchmark_data_smithi038_77863_object23578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:07:05.198 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.95716683800001, 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063222441000000004, 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.199 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23579 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222049+0000", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.437e-06 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222105+0000", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6329000000000002e-05 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222116+0000", 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1518000000000001e-05 2024-03-04T16:07:05.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222122+0000", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0340000000000002e-06 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222173+0000", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0720999999999998e-05 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222270+0000", 2024-03-04T16:07:05.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6972999999999995e-05 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222399+0000", 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128999 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.223163+0000", 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076355399999999995 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.223487+0000", 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000324729 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285180+0000", 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061692898000000003 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285240+0000", 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9389999999999999e-05 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285270+0000", 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9859e-05 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24064 7.1 7:8a6e839d:::benchmark_data_smithi038_77863_object24063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.68173798700001, 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12912359700000001, 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.205 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24064 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.206 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497478+0000", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.956e-06 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497544+0000", 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6097000000000001e-05 2024-03-04T16:07:05.207 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497559+0000", 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4935e-05 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497568+0000", 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9970000000000004e-06 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.208 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497637+0000", 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8853e-05 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497782+0000", 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014473599999999999 2024-03-04T16:07:05.209 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497906+0000", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124091 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499051+0000", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011451409999999999 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499189+0000", 2024-03-04T16:07:05.210 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013806699999999999 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626465+0000", 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12727575699999999 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626544+0000", 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.907e-05 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.211 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626600+0000", 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5896999999999997e-05 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24066 7.1 7:873d823e:::benchmark_data_smithi038_77863_object24065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.680068405, 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.130573199, 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.212 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24066 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499147+0000", 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 0.256982 (41c9) 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.274e-06 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499199+0000", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2524000000000001e-05 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499208+0000", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6740000000000003e-06 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499214+0000", 2024-03-04T16:07:05.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5999999999999997e-06 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499248+0000", 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4187999999999999e-05 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499344+0000", 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6173000000000002e-05 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499488+0000", 2024-03-04T16:07:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014452200000000001 2024-03-04T16:07:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.500600+0000", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011114740000000001 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.501582+0000", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00098159099999999997 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629612+0000", 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12803046100000001 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629673+0000", 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1291000000000007e-05 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629719+0000", 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5426999999999997e-05 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24076 7.1 7:802e523e:::benchmark_data_smithi038_77863_object24075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.67529570799999, 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12599276100000001, 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24076 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.221 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503920+0000", 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4190000000000001e-06 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504014+0000", 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4440999999999999e-05 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504025+0000", 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0706999999999999e-05 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504032+0000", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.002e-06 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504086+0000", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4079999999999997e-05 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504200+0000", 2024-03-04T16:07:05.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114207 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504319+0000", 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011859899999999999 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505150+0000", 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083150499999999996 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505431+0000", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028036800000000002 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629771+0000", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12434076600000001 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629841+0000", 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9821999999999997e-05 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629911+0000", 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9845000000000006e-05 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24080 7.1 7:85c22b13:::benchmark_data_smithi038_77863_object24079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.66851722200001, 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11934697599999999, 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.227 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24080 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.228 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510699+0000", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0370000000000001e-06 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510740+0000", 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1344000000000003e-05 2024-03-04T16:07:05.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510753+0000", 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3010000000000001e-05 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510760+0000", 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3000000000000004e-06 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510820+0000", 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9786e-05 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510959+0000", 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013839899999999999 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511076+0000", 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117656 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511948+0000", 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00087144999999999996 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.512095+0000", 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147682 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629957+0000", 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11786121500000001 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630007+0000", 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0105000000000002e-05 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630044+0000", 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6992000000000001e-05 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.233 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:31962 7.1 7:8e401fd9:::benchmark_data_smithi038_77863_object31961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:01.619438+0000", 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 123.55977636900001, 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.057907308999999997, 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 31962 2024-03-04T16:07:05.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619438+0000", 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619438+0000", 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619439+0000", 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.654e-06 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619496+0000", 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.711e-05 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619509+0000", 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2334000000000001e-05 2024-03-04T16:07:05.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619515+0000", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0619999999999997e-06 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619563+0000", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8649000000000001e-05 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619655+0000", 2024-03-04T16:07:05.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1483999999999993e-05 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.619747+0000", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2282999999999997e-05 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.620864+0000", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001116683 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.621025+0000", 2024-03-04T16:07:05.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016106599999999999 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.677267+0000", 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.056242453999999997 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.677315+0000", 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8139000000000001e-05 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:01.677345+0000", 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9391e-05 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38263 7.1 7:894d19c1:::benchmark_data_smithi038_77863_object38262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 120.005729133, 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067151086999999998, 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.240 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38263 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173487+0000", 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8250000000000001e-06 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173547+0000", 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9956e-05 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173558+0000", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1263e-05 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173564+0000", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9290000000000003e-06 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173616+0000", 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2658999999999998e-05 2024-03-04T16:07:05.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173723+0000", 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106154 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173821+0000", 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8696000000000005e-05 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.174908+0000", 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010863069999999999 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.175260+0000", 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035221599999999998 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240529+0000", 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065269674999999999 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240588+0000", 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8378000000000002e-05 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240636+0000", 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8028999999999997e-05 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38267 7.1 7:8d7ce7cb:::benchmark_data_smithi038_77863_object38266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:05.246 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 120.000943789, 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064004921000000006, 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38267 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178273+0000", 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178332+0000", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8739999999999997e-05 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178365+0000", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3142999999999998e-05 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178380+0000", 2024-03-04T16:07:05.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5199e-05 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178504+0000", 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012342000000000001 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178717+0000", 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000213258 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178801+0000", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3704999999999999e-05 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.179805+0000", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010037799999999999 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.179905+0000", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100589 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242182+0000", 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062276435999999998 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242237+0000", 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4957000000000003e-05 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242275+0000", 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8544000000000002e-05 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.252 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:69219 7.1 7:83220939:::benchmark_data_smithi038_77863_object69218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:27.579110+0000", 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.600103816000001, 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061912134000000001, 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69219 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.253 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579110+0000", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579110+0000", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579111+0000", 2024-03-04T16:07:05.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.066e-06 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579175+0000", 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3880000000000005e-05 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579187+0000", 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.167e-05 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.579194+0000", 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3749999999999997e-06 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582876+0000", 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036820860000000002 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.583026+0000", 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015029400000000001 2024-03-04T16:07:05.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.583191+0000", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016444800000000001 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.587697+0000", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045062319999999998 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.588272+0000", 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057521999999999998 2024-03-04T16:07:05.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.640861+0000", 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.052588193999999998 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.640954+0000", 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3602000000000003e-05 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.641022+0000", 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8066999999999997e-05 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:69224 7.1 7:8b3514e8:::benchmark_data_smithi038_77863_object69223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:27.582871+0000", 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.596342437999994, 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.060678604999999997, 2024-03-04T16:07:05.260 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 69224 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582871+0000", 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582871+0000", 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582874+0000", 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.091e-06 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582908+0000", 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4853e-05 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582919+0000", 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0723999999999999e-05 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.582928+0000", 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5129999999999996e-06 2024-03-04T16:07:05.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.585215+0000", 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022871699999999998 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.585386+0000", 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017154799999999999 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.585546+0000", 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015924400000000001 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.587877+0000", 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00233132 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.588348+0000", 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00047066600000000001 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.643475+0000", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.055127150999999999 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.643509+0000", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4096e-05 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:27.643550+0000", 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1229000000000001e-05 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.266 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:82966 7.1 7:8f76a7d4:::benchmark_data_smithi038_77863_object82965:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:39.667924+0000", 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 85.511290324000001, 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058044217000000002, 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82966 2024-03-04T16:07:05.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667924+0000", 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667924+0000", 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667926+0000", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9580000000000001e-06 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667965+0000", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9903000000000001e-05 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667979+0000", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3268000000000001e-05 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.667990+0000", 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1545999999999999e-05 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668039+0000", 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8862000000000001e-05 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668152+0000", 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112732 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668280+0000", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012860700000000001 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.673742+0000", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054614279999999999 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674194+0000", 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045250599999999998 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.725847+0000", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051652121000000002 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.725901+0000", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4907000000000001e-05 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.725968+0000", 2024-03-04T16:07:05.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6378999999999996e-05 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:82967 7.1 7:8676c0e4:::benchmark_data_smithi038_77863_object82966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:39.668104+0000", 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 85.511109574000002, 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.058136924999999999, 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82967 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668104+0000", 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668104+0000", 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668106+0000", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2550000000000001e-06 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668233+0000", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126978 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668243+0000", 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0325e-05 2024-03-04T16:07:05.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.668251+0000", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5970000000000003e-06 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670989+0000", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027383989999999999 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.671116+0000", 2024-03-04T16:07:05.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012687899999999999 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.671301+0000", 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000184897 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.673899+0000", 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025980339999999999 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674278+0000", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037974599999999998 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.726154+0000", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.051875943000000001 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.726205+0000", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0890000000000002e-05 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.726241+0000", 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5982000000000002e-05 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83920 7.1 7:8d8feee9:::benchmark_data_smithi038_77863_object83919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.525995420000001, 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10767995700000001, 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83920 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653220+0000", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.545e-06 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653270+0000", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9568999999999999e-05 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653281+0000", 2024-03-04T16:07:05.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1192999999999999e-05 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653296+0000", 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4987999999999999e-05 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653345+0000", 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8775e-05 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653480+0000", 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013508899999999999 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653647+0000", 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167004 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656243+0000", 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025961299999999999 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656442+0000", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199464 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760825+0000", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.104383084 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760863+0000", 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7704999999999999e-05 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760898+0000", 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5410999999999997e-05 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83925 7.1 7:87a9c3a2:::benchmark_data_smithi038_77863_object83924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 84.522644018999998, 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.107333128, 2024-03-04T16:07:05.285 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83925 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656572+0000", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.778e-06 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656615+0000", 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3161000000000002e-05 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656626+0000", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1065e-05 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656632+0000", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7389999999999996e-06 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656674+0000", 2024-03-04T16:07:05.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656764+0000", 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0019999999999995e-05 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656883+0000", 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119466 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659071+0000", 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021878219999999999 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659140+0000", 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8551000000000001e-05 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763805+0000", 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10466587199999999 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763859+0000", 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3298999999999999e-05 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763903+0000", 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4301999999999997e-05 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.291 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84327 7.1 7:8c4037d9:::benchmark_data_smithi038_77863_object84326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.984791001999994, 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15446056899999999, 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84327 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194425+0000", 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194465+0000", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9663000000000003e-05 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194479+0000", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4025999999999999e-05 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194488+0000", 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8810000000000002e-06 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194551+0000", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2869999999999999e-05 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194677+0000", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012619000000000001 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194823+0000", 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145477 2024-03-04T16:07:05.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197906+0000", 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030836819999999999 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197926+0000", 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355999999999999e-05 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348763+0000", 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15083737699999999 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348834+0000", 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1464999999999994e-05 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348883+0000", 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9094000000000001e-05 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.297 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84339 7.1 7:81128b26:::benchmark_data_smithi038_77863_object84338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 83.963196800999995, 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132991251, 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84339 2024-03-04T16:07:05.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216020+0000", 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6180000000000002e-06 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216053+0000", 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3401000000000001e-05 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216067+0000", 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3816e-05 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216075+0000", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6839999999999996e-06 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217281+0000", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001206181 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217443+0000", 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162063 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217620+0000", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017687900000000001 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219824+0000", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022046380000000001 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219965+0000", 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014102400000000001 2024-03-04T16:07:05.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348936+0000", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128970469 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348971+0000", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5068999999999999e-05 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.349008+0000", 2024-03-04T16:07:05.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7409e-05 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:103907 7.1 7:819e1ab5:::benchmark_data_smithi038_77863_object103906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.273285377999997, 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.066058037999999999, 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 103907 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905931+0000", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0420000000000001e-06 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906024+0000", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3833999999999996e-05 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906037+0000", 2024-03-04T16:07:05.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2571e-05 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906045+0000", 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7449999999999995e-06 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906078+0000", 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3556999999999997e-05 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906187+0000", 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108799 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906335+0000", 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014748300000000001 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.908898+0000", 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002563358 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.908915+0000", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7507999999999999e-05 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971911+0000", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062995491000000001 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971954+0000", 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3084999999999998e-05 2024-03-04T16:07:05.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971987+0000", 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2564999999999997e-05 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:103910 7.1 7:8da212fe:::benchmark_data_smithi038_77863_object103909:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:02.909363+0000", 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 62.269850638000001, 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062720855000000006, 2024-03-04T16:07:05.310 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 103910 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909363+0000", 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909363+0000", 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909365+0000", 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.482e-06 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:05.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909422+0000", 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7318000000000002e-05 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909432+0000", 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5959999999999993e-06 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909440+0000", 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8569999999999995e-06 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909475+0000", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5348000000000001e-05 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909557+0000", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2490000000000005e-05 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.909653+0000", 2024-03-04T16:07:05.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5412000000000001e-05 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.911691+0000", 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020377889999999999 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.911921+0000", 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022998499999999999 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:05.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.972021+0000", 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.060100594 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.972056+0000", 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5108999999999999e-05 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.972084+0000", 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7875e-05 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.316 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:05.317 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:05.317 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 982 op/s 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3265715682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: from='client.34856 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:05.317 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]: dispatch 2024-03-04T16:07:05.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.344 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:05.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:05.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:05.485 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 982 op/s 2024-03-04T16:07:05.485 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3265715682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.485 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: from='client.34856 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.485 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]: dispatch 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 956 B/s rd, 0 B/s wr, 982 op/s 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3265715682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: from='client.34856 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:05.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]: dispatch 2024-03-04T16:07:05.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:05.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:05.821 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.846 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:05.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:05.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:05.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:06.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.098 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:06.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:06.199 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:06.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:06.335 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:06.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:06.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:06.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[38113]: from='client.34856 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]': finished 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[38113]: osdmap e127: 8 total, 8 up, 8 in 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[38113]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s 2024-03-04T16:07:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:06.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[29779]: from='client.34856 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:06.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]': finished 2024-03-04T16:07:06.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[29779]: osdmap e127: 8 total, 8 up, 8 in 2024-03-04T16:07:06.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:06 smithi038 ceph-mon[29779]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s 2024-03-04T16:07:06.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:06 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:06 smithi176 ceph-mon[33619]: from='client.34856 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/798097391' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 0.2569816671582034}]': finished 2024-03-04T16:07:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:06 smithi176 ceph-mon[33619]: osdmap e127: 8 total, 8 up, 8 in 2024-03-04T16:07:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:06 smithi176 ceph-mon[33619]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s 2024-03-04T16:07:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.611 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:06.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:06.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:06.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:07:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:06.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:06.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:06.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:06.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:06.978 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:06.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:06.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:07.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:07.121 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:07.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:07.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:07 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:07 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:07 smithi176 ceph-mon[33619]: osdmap e128: 8 total, 8 up, 8 in 2024-03-04T16:07:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:07.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:07.364 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:07.399 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[29779]: osdmap e128: 8 total, 8 up, 8 in 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:07.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[38113]: osdmap e128: 8 total, 8 up, 8 in 2024-03-04T16:07:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:07.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:07.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:07.541 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:07.605 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:07:07.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:07.661 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:07:07.678 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:07.700 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:07:07.744 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:07:07.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:07.787 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:07:07.822 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:07.828 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:07:07.870 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:07:07.910 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:07:07.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:07:07.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:07.969 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:07.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:07.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:08.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:08.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:08.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:08.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:08.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:08.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:08.481 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 126'10237 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 126'10231 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: osdmap e129: 8 total, 8 up, 8 in 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10160 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10268 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 126'12333 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 126'10202 2024-03-04T16:07:08.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 126'10124 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 778 op/s; 54 B/s, 1 objects/s recovering 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1789782742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 126'10237 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 126'10231 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: osdmap e129: 8 total, 8 up, 8 in 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10160 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10268 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 126'12333 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 126'10202 2024-03-04T16:07:08.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 126'10124 2024-03-04T16:07:08.484 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 778 op/s; 54 B/s, 1 objects/s recovering 2024-03-04T16:07:08.484 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:08 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1789782742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.527 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:08.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:08.586 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 126'10237 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 126'10231 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: osdmap e129: 8 total, 8 up, 8 in 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10160 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 126'10268 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 126'12333 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 126'10202 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 126'10124 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 778 op/s; 54 B/s, 1 objects/s recovering 2024-03-04T16:07:08.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:08 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1789782742' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:08.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:08.674 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:08.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:08.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:08.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:08.968 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:08.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:09.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:09.102 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:07:09.103 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:07:09.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:09.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:09.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:09.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:07:09.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.405 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:09.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:09.424 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:09 smithi176 ceph-mon[33619]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:07:09.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:09.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:09.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:09.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:09 smithi038 ceph-mon[29779]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:07:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:09 smithi038 ceph-mon[38113]: osdmap e130: 8 total, 8 up, 8 in 2024-03-04T16:07:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:09.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.832 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:09.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:09.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:09.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:09.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:10.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.113 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:10.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:10.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.240 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:10.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:10.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:10.379 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:10.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:10.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:10 smithi038 ceph-mon[29779]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:07:10.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:10 smithi038 ceph-mon[29779]: pgmap v168: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 31888/218637 objects misplaced (14.585%); 737 KiB/s, 23 objects/s recovering 2024-03-04T16:07:10.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:10 smithi038 ceph-mon[38113]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:07:10.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:10 smithi038 ceph-mon[38113]: pgmap v168: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 31888/218637 objects misplaced (14.585%); 737 KiB/s, 23 objects/s recovering 2024-03-04T16:07:10.523 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:10.532 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:10 smithi176 ceph-mon[33619]: osdmap e131: 8 total, 8 up, 8 in 2024-03-04T16:07:10.532 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:10 smithi176 ceph-mon[33619]: pgmap v168: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 629 op/s; 31888/218637 objects misplaced (14.585%); 737 KiB/s, 23 objects/s recovering 2024-03-04T16:07:10.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.661 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:10.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:10.762 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:10.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:10.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:07:10.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:07:10.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:10.955 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:10.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:10.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:11.057 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.090 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:11.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:11.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:11.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:11.250 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:11.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.378 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:11.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:11.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:11.511 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:11.524 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:11.524 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:11.524 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:11 smithi176 ceph-mon[33619]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:07:11.524 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:11 smithi176 ceph-mon[33619]: osdmap e132: 8 total, 8 up, 8 in 2024-03-04T16:07:11.570 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:11.570 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[29779]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[29779]: osdmap e132: 8 total, 8 up, 8 in 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[38113]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:07:11.571 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:11 smithi038 ceph-mon[38113]: osdmap e132: 8 total, 8 up, 8 in 2024-03-04T16:07:11.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:11.653 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:11.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:11.770 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.817 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:11.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:11.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:11.970 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:11.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:12.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:12.106 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:12.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:12.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:12.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:12.304 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:07:12.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:12.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.398 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:12.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[29779]: pgmap v170: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 31888/218637 objects misplaced (14.585%); 711 KiB/s, 23 objects/s recovering 2024-03-04T16:07:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3470345006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[29779]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[38113]: pgmap v170: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 31888/218637 objects misplaced (14.585%); 711 KiB/s, 23 objects/s recovering 2024-03-04T16:07:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3470345006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:12 smithi038 ceph-mon[38113]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:12.505 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:12 smithi176 ceph-mon[33619]: pgmap v170: 148 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 1 active+undersized+remapped, 140 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 607 op/s; 31888/218637 objects misplaced (14.585%); 711 KiB/s, 23 objects/s recovering 2024-03-04T16:07:12.505 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3470345006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.506 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:12 smithi176 ceph-mon[33619]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:07:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.540 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:12.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:12.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:12.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.804 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:12.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:12.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:12.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:12.932 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:12.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:13.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:13.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:13.124 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:07:13.125 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:07:13.125 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-04T16:07:13.125 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.778322 2024-03-04T16:07:13.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 0 0.7783216735084586 2024-03-04T16:07:13.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.197 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:13.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:13.329 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:13.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.466 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:13.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:13.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:13 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:13 smithi176 ceph-mon[33619]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:07:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:13 smithi176 ceph-mon[33619]: osdmap e133: 8 total, 8 up, 8 in 2024-03-04T16:07:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:13.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:13.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:13.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[29779]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:07:13.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[29779]: osdmap e133: 8 total, 8 up, 8 in 2024-03-04T16:07:13.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:13.738 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:13.739 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:13.739 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[38113]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:07:13.739 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[38113]: osdmap e133: 8 total, 8 up, 8 in 2024-03-04T16:07:13.739 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:13.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:13.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:13.884 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:13.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:13.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:14.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:14.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.161 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:14.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:14.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:14.293 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:14.379 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:14.394 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:14.426 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:14.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:14.540 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:07:14.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[29779]: pgmap v172: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 997 B/s rd, 0 B/s wr, 928 op/s; 26417/204363 objects misplaced (12.927%); 42 MiB/s, 678 objects/s recovering 2024-03-04T16:07:14.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:14.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:14.541 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[38113]: pgmap v172: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 997 B/s rd, 0 B/s wr, 928 op/s; 26417/204363 objects misplaced (12.927%); 42 MiB/s, 678 objects/s recovering 2024-03-04T16:07:14.541 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:14.542 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:14 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:14.567 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:14.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:14.606 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:07:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:14 smithi176 ceph-mon[33619]: pgmap v172: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 997 B/s rd, 0 B/s wr, 928 op/s; 26417/204363 objects misplaced (12.927%); 42 MiB/s, 678 objects/s recovering 2024-03-04T16:07:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:14 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:14 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:14.646 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:07:14.669 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:14.698 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:14.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:07:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:14.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:07:14.794 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:07:14.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:14.838 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:07:14.845 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:14.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:07:14.889 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:07:14.889 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:07:14.889 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:07:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:07:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:07:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:07:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:07:14.890 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:07:14.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:14.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:14.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:15.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:15.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.281 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:15.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:15 smithi176 ceph-mon[33619]: osdmap e134: 8 total, 8 up, 8 in 2024-03-04T16:07:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3455614174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:15 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:07:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:15 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:07:15.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:07:15.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:15.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:15.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.591 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:15.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:15.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[29779]: osdmap e134: 8 total, 8 up, 8 in 2024-03-04T16:07:15.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3455614174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[38113]: osdmap e134: 8 total, 8 up, 8 in 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3455614174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:07:15.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:15 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:07:15.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.727 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:15.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:15.767 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:15.767 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:07:15.767 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:07:15.767 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:07:15.767 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:15.827 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:15.866 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:15.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:15.967 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:16.000 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:16.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:16.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:16.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:16.281 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:16.383 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:16 smithi038 ceph-mon[29779]: pgmap v174: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:16.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:16 smithi038 ceph-mon[38113]: pgmap v174: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:16.514 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:16 smithi176 ceph-mon[33619]: pgmap v174: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:16.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:16.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:07:16.564 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:16.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.706 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:16.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:16.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:16.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:16.939 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:16.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:17.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:17.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:17.161 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:17.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:17.267 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:17.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[29779]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:17.342 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:17.343 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:07:17.343 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:07:17.343 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:17 smithi038 ceph-mon[38113]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:07:17.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:17 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:17.344 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:17 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:17.344 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:17 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:07:17.344 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:17 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:07:17.344 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:17 smithi176 ceph-mon[33619]: osdmap e135: 8 total, 8 up, 8 in 2024-03-04T16:07:17.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.440 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:17.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:17.541 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:17.574 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:17.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:17.715 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:17.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:17.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:17.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:17.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:18.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:18.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:18.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:18.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.281 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:18.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:18.342 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.0 primary-affinity to 0.778322 (8510082) 2024-03-04T16:07:18.381 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[29779]: pgmap v176: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3589557857' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[29779]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[38113]: pgmap v176: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:18.418 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:18.419 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3589557857' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.419 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:18 smithi038 ceph-mon[38113]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.438 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:18.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:18.525 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:18 smithi176 ceph-mon[33619]: pgmap v176: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 793 op/s; 26417/204363 objects misplaced (12.927%); 36 MiB/s, 579 objects/s recovering 2024-03-04T16:07:18.525 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:18 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:07:18.526 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3589557857' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.526 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:18 smithi176 ceph-mon[33619]: from='client.34880 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]: dispatch 2024-03-04T16:07:18.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:18.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:07:18.602 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:18.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:18.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:18.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:18.918 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:18.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:19.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.053 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:19.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:19.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:19.190 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:19.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:19.296 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:8983 7.0 7:03998ed9:::benchmark_data_smithi038_77863_object8982:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:51.005397+0000", 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 148.285332598, 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.073157679000000003, 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.297 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 8983 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005397+0000", 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005397+0000", 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005399+0000", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0090000000000002e-06 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005505+0000", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106234 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005519+0000", 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3791999999999999e-05 2024-03-04T16:07:19.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005527+0000", 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2600000000000005e-06 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005595+0000", 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7737999999999994e-05 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005729+0000", 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013417 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005856+0000", 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126403 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.006427+0000", 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00057166499999999996 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.007261+0000", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083325999999999995 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.078473+0000", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071212514000000005 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.078511+0000", 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7589000000000002e-05 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.078555+0000", 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4045000000000003e-05 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:22852 7.0 7:0b12272d:::benchmark_data_smithi038_77863_object22851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:56.771694+0000", 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 142.51903550500001, 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069639996999999995, 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.303 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 22852 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771694+0000", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771694+0000", 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771696+0000", 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6750000000000001e-06 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771789+0000", 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2859e-05 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771799+0000", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.077e-05 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771805+0000", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9359999999999996e-06 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771852+0000", 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6848000000000003e-05 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.771939+0000", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6619999999999994e-05 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.772027+0000", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8227999999999996e-05 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.772811+0000", 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078396399999999995 2024-03-04T16:07:19.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.773869+0000", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001058135 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.841223+0000", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067353352000000005 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.841279+0000", 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6758000000000001e-05 2024-03-04T16:07:19.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.841334+0000", 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4852000000000003e-05 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23137 7.0 7:04f8d3f5:::benchmark_data_smithi038_77863_object23136:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:56.959126+0000", 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 142.33160348499999, 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.077224854999999995, 2024-03-04T16:07:19.309 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23137 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959126+0000", 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959126+0000", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959128+0000", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.075e-06 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959166+0000", 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7626000000000002e-05 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959177+0000", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1144999999999999e-05 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959185+0000", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2840000000000006e-06 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959247+0000", 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1730999999999996e-05 2024-03-04T16:07:19.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959385+0000", 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000137596 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.959514+0000", 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012928899999999999 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.960444+0000", 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00092960000000000004 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961060+0000", 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061608699999999997 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.036232+0000", 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075172569999999994 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.036312+0000", 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0055999999999994e-05 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.036351+0000", 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8795999999999999e-05 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.315 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23143 7.0 7:03e5f237:::benchmark_data_smithi038_77863_object23142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:56.960928+0000", 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 142.32980163100001, 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.078527861000000004, 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23143 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.960928+0000", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.960928+0000", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.960930+0000", 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8300000000000001e-06 2024-03-04T16:07:19.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961000+0000", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0327999999999995e-05 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961015+0000", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4491999999999999e-05 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961023+0000", 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4069999999999993e-06 2024-03-04T16:07:19.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961083+0000", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9570000000000001e-05 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961172+0000", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8973000000000003e-05 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961261+0000", 2024-03-04T16:07:19.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9046000000000001e-05 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.961937+0000", 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00067597399999999998 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:56.962391+0000", 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00045458599999999999 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.039310+0000", 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.076918713 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.039402+0000", 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2048999999999999e-05 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.039456+0000", 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3893000000000001e-05 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.321 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24056 7.0 7:0d6c155e:::benchmark_data_smithi038_77863_object24055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.494975+0000", 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 141.79575456800001, 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13333609099999999, 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24056 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.494975+0000", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.494975+0000", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.494977+0000", 2024-03-04T16:07:19.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8700000000000001e-06 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495041+0000", 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4381999999999994e-05 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495055+0000", 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3405e-05 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495062+0000", 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3679999999999996e-06 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495114+0000", 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2330000000000002e-05 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495222+0000", 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010780099999999999 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495366+0000", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014389700000000001 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496458+0000", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001092277 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.498602+0000", 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021431140000000002 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628138+0000", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12953609399999999 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628254+0000", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116069 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628311+0000", 2024-03-04T16:07:19.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7484000000000001e-05 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24062 7.0 7:0d65af63:::benchmark_data_smithi038_77863_object24061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.497143+0000", 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 141.793586576, 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13135572700000001, 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24062 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497143+0000", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497143+0000", 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497145+0000", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.829e-06 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497204+0000", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9460999999999999e-05 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497220+0000", 2024-03-04T16:07:19.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5208000000000001e-05 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497227+0000", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5039999999999996e-06 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497295+0000", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8174000000000002e-05 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497405+0000", 2024-03-04T16:07:19.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110142 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497490+0000", 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4470999999999998e-05 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.498617+0000", 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001127 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499494+0000", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00087669199999999999 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628379+0000", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128885892 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628459+0000", 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9318000000000002e-05 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628499+0000", 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0036000000000001e-05 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.334 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24074 7.0 7:0038c09d:::benchmark_data_smithi038_77863_object24073:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.503138+0000", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 141.787591324, 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.125529534, 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24074 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503138+0000", 2024-03-04T16:07:19.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503138+0000", 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503140+0000", 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7209999999999999e-06 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503188+0000", 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7676999999999998e-05 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503200+0000", 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2238999999999999e-05 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503206+0000", 2024-03-04T16:07:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8030000000000002e-06 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503254+0000", 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7901000000000001e-05 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503389+0000", 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013498899999999999 2024-03-04T16:07:19.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503495+0000", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010676 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504501+0000", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010056189999999999 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504799+0000", 2024-03-04T16:07:19.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029804199999999999 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628581+0000", 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12378196700000001 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628632+0000", 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1150999999999999e-05 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628668+0000", 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5664999999999998e-05 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38245 7.0 7:02162b20:::benchmark_data_smithi038_77863_object38244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.163118+0000", 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.127611712, 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079091993999999999, 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.342 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38245 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163118+0000", 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163118+0000", 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163120+0000", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.091e-06 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163213+0000", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3121000000000005e-05 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163224+0000", 2024-03-04T16:07:19.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0898e-05 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163234+0000", 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6840000000000008e-06 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163312+0000", 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8621000000000005e-05 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163456+0000", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143468 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163616+0000", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016040299999999999 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167213+0000", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035963829999999999 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.170314+0000", 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031015779999999998 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242112+0000", 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071798031999999998 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242166+0000", 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3967e-05 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242210+0000", 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3748000000000002e-05 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38246 7.0 7:0a57c1e6:::benchmark_data_smithi038_77863_object38245:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.163439+0000", 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.1272908, 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.078885970999999999, 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.348 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38246 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163439+0000", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163439+0000", 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163440+0000", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4309999999999999e-06 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163539+0000", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8456999999999996e-05 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163557+0000", 2024-03-04T16:07:19.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8427000000000002e-05 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163565+0000", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4259999999999997e-06 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.164937+0000", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013727539999999999 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.165068+0000", 2024-03-04T16:07:19.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013045799999999999 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.165205+0000", 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013738499999999999 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167217+0000", 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020117479999999998 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.170357+0000", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003140332 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242255+0000", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071897481999999999 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242294+0000", 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9499999999999998e-05 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242325+0000", 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0571000000000003e-05 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38247 7.0 7:03bdc583:::benchmark_data_smithi038_77863_object38246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.163601+0000", 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.12712907900001, 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.078873328000000006, 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.354 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38247 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163601+0000", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163601+0000", 2024-03-04T16:07:19.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163607+0000", 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2929999999999997e-06 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163658+0000", 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1134999999999997e-05 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163666+0000", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1049999999999995e-06 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.163673+0000", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7499999999999997e-06 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.166198+0000", 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025250020000000001 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.166355+0000", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000156978 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.166456+0000", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100645 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167361+0000", 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090519899999999998 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.172404+0000", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050436020000000003 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242374+0000", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069969931999999999 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242437+0000", 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2562000000000002e-05 2024-03-04T16:07:19.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242474+0000", 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7125000000000001e-05 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38251 7.0 7:07cd144a:::benchmark_data_smithi038_77863_object38250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.167576+0000", 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.123153374, 2024-03-04T16:07:19.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075024575999999996, 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38251 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167576+0000", 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167576+0000", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167578+0000", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6619999999999999e-06 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167644+0000", 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6167000000000005e-05 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167657+0000", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3237999999999999e-05 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167666+0000", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9420000000000001e-06 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167728+0000", 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1408000000000006e-05 2024-03-04T16:07:19.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.167854+0000", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000126749 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.168002+0000", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147202 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.169022+0000", 2024-03-04T16:07:19.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010199199999999999 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.172446+0000", 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034239579999999999 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242512+0000", 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070066635000000002 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242558+0000", 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5884999999999999e-05 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242601+0000", 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2809999999999998e-05 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38258 7.0 7:040aed7c:::benchmark_data_smithi038_77863_object38257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.171111+0000", 2024-03-04T16:07:19.366 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 134.11961846899999, 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.071797156000000001, 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38258 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171111+0000", 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171111+0000", 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171113+0000", 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4559999999999999e-06 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171157+0000", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4437999999999997e-05 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171168+0000", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0616e-05 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171174+0000", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.84e-06 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171224+0000", 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0878999999999999e-05 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171358+0000", 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013321599999999999 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.171450+0000", 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1842000000000006e-05 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.172495+0000", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010456770000000001 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173011+0000", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00051584099999999996 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242797+0000", 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.069786335000000005 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242860+0000", 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2439000000000003e-05 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242908+0000", 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8576999999999999e-05 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.372 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:76057 7.0 7:0b139103:::benchmark_data_smithi038_77863_object76056:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:33.373199+0000", 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 105.917530773, 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.086516784999999999, 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 76057 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373199+0000", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373199+0000", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373203+0000", 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5369999999999999e-06 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373264+0000", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0537e-05 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373278+0000", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4005999999999999e-05 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373284+0000", 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0769999999999996e-06 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373370+0000", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5968000000000001e-05 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373517+0000", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014704 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.373666+0000", 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014873999999999999 2024-03-04T16:07:19.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.375921+0000", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022554599999999999 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.377293+0000", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0013718840000000001 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.459615+0000", 2024-03-04T16:07:19.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.082321621999999997 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.459675+0000", 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0460000000000001e-05 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:33.459716+0000", 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0454000000000003e-05 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.379 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:19.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:19.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:19 smithi038 ceph-mon[29779]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]': finished 2024-03-04T16:07:19.380 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:19 smithi038 ceph-mon[29779]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:07:19.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:19 smithi038 ceph-mon[38113]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]': finished 2024-03-04T16:07:19.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:19 smithi038 ceph-mon[38113]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:82970 7.0 7:02fe7a43:::benchmark_data_smithi038_77863_object82969:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:39.670893+0000", 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 99.619836394000004, 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.068805359999999996, 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.381 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82970 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670893+0000", 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670893+0000", 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670896+0000", 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3999999999999999e-06 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670934+0000", 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8170000000000002e-05 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670952+0000", 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7966999999999999e-05 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.670961+0000", 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7409999999999998e-06 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.671011+0000", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0819000000000003e-05 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.671107+0000", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5241000000000005e-05 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.671213+0000", 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106551 2024-03-04T16:07:19.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.673975+0000", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027614330000000002 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.677218+0000", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003243115 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739429+0000", 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062211339999999997 2024-03-04T16:07:19.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739583+0000", 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000154418 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739699+0000", 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115165 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:82973 7.0 7:01f22558:::benchmark_data_smithi038_77863_object82972:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:39.674364+0000", 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 99.616365473000002, 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065550526999999997, 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 82973 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674364+0000", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674364+0000", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674367+0000", 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5390000000000002e-06 2024-03-04T16:07:19.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674416+0000", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9629999999999997e-05 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674430+0000", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3399e-05 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674437+0000", 2024-03-04T16:07:19.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0180000000000004e-06 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674484+0000", 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.761e-05 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674583+0000", 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8199e-05 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.674678+0000", 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5054000000000002e-05 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.676567+0000", 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018897339999999999 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.677317+0000", 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074960700000000001 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739746+0000", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062429088000000001 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739875+0000", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012918200000000001 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:39.739915+0000", 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9467e-05 2024-03-04T16:07:19.393 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83924 7.0 7:0df3cca0:::benchmark_data_smithi038_77863_object83923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.656491+0000", 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.634238221999993, 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114364142, 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.394 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83924 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656491+0000", 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656491+0000", 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656493+0000", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.471e-06 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656553+0000", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9797999999999999e-05 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656562+0000", 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7389999999999994e-06 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656568+0000", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5509999999999999e-06 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656609+0000", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0794000000000003e-05 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656704+0000", 2024-03-04T16:07:19.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5490999999999998e-05 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656811+0000", 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106628 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659069+0000", 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022585119999999998 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.660166+0000", 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010960830000000001 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.770785+0000", 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11061977100000001 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.770826+0000", 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0272000000000003e-05 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.770856+0000", 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0032e-05 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84323 7.0 7:0a2f94c7:::benchmark_data_smithi038_77863_object84322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.190402+0000", 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 98.100328125999994, 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.160349727, 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.400 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84323 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190402+0000", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190402+0000", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190404+0000", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2419999999999999e-06 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190448+0000", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4076999999999998e-05 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.402 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190463+0000", 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4636e-05 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190471+0000", 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.242e-06 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190540+0000", 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9503000000000002e-05 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190669+0000", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128718 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190815+0000", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146379 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.193969+0000", 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003153187 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.404 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227609+0000", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.033640884000000003 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.350656+0000", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.123046207 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.350701+0000", 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4999000000000001e-05 2024-03-04T16:07:19.405 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.350751+0000", 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0652999999999998e-05 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159708 7.0 7:03c4cb35:::benchmark_data_smithi038_77863_object41440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.638225+0000", 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.652505073, 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13804856700000001, 2024-03-04T16:07:19.406 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159708 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638225+0000", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.407 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638225+0000", 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638319+0000", 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4493000000000005e-05 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638326+0000", 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.533e-06 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.408 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638339+0000", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3474e-05 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638350+0000", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0774999999999999e-05 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638406+0000", 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6233999999999997e-05 2024-03-04T16:07:19.409 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638524+0000", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118195 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638630+0000", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010530399999999999 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.641355+0000", 2024-03-04T16:07:19.410 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027250550000000001 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642483+0000", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011285010000000001 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776204+0000", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13372077099999999 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776240+0000", 2024-03-04T16:07:19.411 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6010000000000003e-05 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776273+0000", 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3222000000000001e-05 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159715 7.0 7:0166e3a4:::benchmark_data_smithi038_77863_object41447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.643509+0000", 2024-03-04T16:07:19.412 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 18.647221032000001, 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13308481999999999, 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159715 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643509+0000", 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.413 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643509+0000", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643511+0000", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8989999999999999e-06 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643516+0000", 2024-03-04T16:07:19.414 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8130000000000001e-06 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643530+0000", 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3346999999999999e-05 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643537+0000", 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5120000000000002e-06 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.415 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643586+0000", 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8399000000000002e-05 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643668+0000", 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2372000000000003e-05 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643740+0000", 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2158999999999998e-05 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.416 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.646079+0000", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002338718 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.772389+0000", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12630981299999999 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776540+0000", 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041510560000000002 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.417 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776570+0000", 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0295e-05 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776593+0000", 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3436999999999999e-05 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.418 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1415 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93955426379904 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.187940+0000", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 14.102789228000001, 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0429563850000001, 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1415 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:19.419 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187940+0000", 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187940+0000", 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187943+0000", 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4870000000000001e-06 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.420 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187951+0000", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5769999999999996e-06 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187966+0000", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5536000000000002e-05 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187976+0000", 2024-03-04T16:07:19.421 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5510000000000005e-06 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.188041+0000", 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5209000000000005e-05 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.188047+0000", 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8710000000000002e-06 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.422 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.209779+0000", 2024-03-04T16:07:19.423 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021732673000000001 2024-03-04T16:07:19.423 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.209791+0000", 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1460999999999999e-05 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.221984+0000", 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0121927449999999 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:07:19.424 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.222814+0000", 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083076000000000005 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.225322+0000", 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002507188 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.225415+0000", 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3863999999999994e-05 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:19.425 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.230897+0000", 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054814629999999998 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:19.426 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:19.481 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:19.492 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:19 smithi176 ceph-mon[33619]: from='client.34880 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.7783216735084586}]': finished 2024-03-04T16:07:19.492 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:19 smithi176 ceph-mon[33619]: osdmap e136: 8 total, 8 up, 8 in 2024-03-04T16:07:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.525 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:19.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:19.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:19.661 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:19.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:19.801 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:19.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:19.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:19.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:20.049 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:20.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:20.087 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:20.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.232 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:20.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:20.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:20.382 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:20.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:20.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:20.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:20.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:20.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:20.799 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:20.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:20.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:20.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:21.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:07:21.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:21.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:21.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.326 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:21.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:21.364 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:21 smithi176 ceph-mon[33619]: pgmap v178: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:21.364 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2291946824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.364 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:21 smithi176 ceph-mon[33619]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.369 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:07:21.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[29779]: pgmap v178: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:21.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2291946824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[29779]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[38113]: pgmap v178: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:21.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2291946824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.370 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:21 smithi038 ceph-mon[38113]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:21.427 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.477 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:21.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:21.481 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:21.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:21.632 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:21.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:21.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:21.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:21.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:21.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:22.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:22.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:22.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:22.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:22.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.331 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:22.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:07:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-mon[33619]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-mon[33619]: osdmap e137: 8 total, 8 up, 8 in 2024-03-04T16:07:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:22.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:22.468 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[29779]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[29779]: osdmap e137: 8 total, 8 up, 8 in 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:07:22.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[38113]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:22.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[38113]: osdmap e137: 8 total, 8 up, 8 in 2024-03-04T16:07:22.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:22.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:22.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:22.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:22.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.749 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:22.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:22.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:22.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:22.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:22.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:22.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:23.026 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:23.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:23.153 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:23.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:23.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:23.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:23 smithi176 ceph-mon[33619]: pgmap v180: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:23 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:07:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:23 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:07:23.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:23.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:23.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[29779]: pgmap v180: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:23.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:07:23.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:07:23.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[38113]: pgmap v180: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 707 op/s; 21472/191637 objects misplaced (11.205%); 35 MiB/s, 564 objects/s recovering 2024-03-04T16:07:23.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:07:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:07:23.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:23.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:23.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:23.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:23.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:23.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:23.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:23.810 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:23.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:23.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:24.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:24.149 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:07:24.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 1 2024-03-04T16:07:24.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:24.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:24.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:24.353 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:24.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:07:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:07:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[38113]: osdmap e138: 8 total, 8 up, 8 in 2024-03-04T16:07:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:07:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:07:24.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:24 smithi038 ceph-mon[29779]: osdmap e138: 8 total, 8 up, 8 in 2024-03-04T16:07:24.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:24.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:24 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:07:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:24 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:07:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:24 smithi176 ceph-mon[33619]: osdmap e138: 8 total, 8 up, 8 in 2024-03-04T16:07:24.641 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:24.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:24.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:24.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:24.926 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:24.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:25.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:25.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:25.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:25.145 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:179192 7.c 7:3db05f84:::benchmark_data_smithi038_77863_object60924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:25.134341+0000", 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0091383309999999995, 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0091742600000000001, 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:25.146 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179192 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134341+0000", 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134341+0000", 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134344+0000", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.091e-06 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134350+0000", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3280000000000002e-06 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134368+0000", 2024-03-04T16:07:25.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7961e-05 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134377+0000", 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9779999999999994e-06 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:25.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134430+0000", 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2812999999999997e-05 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134545+0000", 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011539 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.134661+0000", 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011561999999999999 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.137849+0000", 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031881019999999999 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138680+0000", 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083084200000000004 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:179195 7.d 7:b352c3d7:::benchmark_data_smithi038_77863_object60927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:25.138059+0000", 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0054208620000000002, 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055620790000000002, 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179195 2024-03-04T16:07:25.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138059+0000", 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138059+0000", 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138061+0000", 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.854e-06 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138067+0000", 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2149999999999998e-06 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138082+0000", 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4745e-05 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138090+0000", 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5110000000000003e-06 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:25.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138197+0000", 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010690499999999999 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138340+0000", 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014328 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.138487+0000", 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000146546 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.141224+0000", 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027365549999999999 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:179198 7.7 7.420f8457 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-04T16:07:25.157 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:25.143324+0000", 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.00015604799999999999, 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035189299999999999, 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "queued for pg", 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179198 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:25.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143324+0000", 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143324+0000", 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143327+0000", 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2839999999999998e-06 2024-03-04T16:07:25.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143334+0000", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0049999999999998e-06 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143346+0000", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1769e-05 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.143354+0000", 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2519999999999999e-06 2024-03-04T16:07:25.160 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:179176 7.d 7:b9374c62:::benchmark_data_smithi038_77863_object60908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:25.122308+0000", 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.021171318000000001, 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.021413537, 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:25.161 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179176 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122308+0000", 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122308+0000", 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:25.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122311+0000", 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6019999999999998e-06 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122319+0000", 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2570000000000007e-06 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122335+0000", 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5404e-05 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122351+0000", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5871000000000001e-05 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122407+0000", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6371e-05 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122520+0000", 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112592 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.122625+0000", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010567 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.126185+0000", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035601370000000001 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.129073+0000", 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028875979999999999 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.165 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-03-04T16:07:25.166 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 4 2024-03-04T16:07:25.166 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:25.181 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:25.217 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:25.264 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: pgmap v182: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 16645/177813 objects misplaced (9.361%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:07:25.264 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:07:25.264 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:07:25.265 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:07:25.265 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:07:25.265 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:25 smithi176 ceph-mon[33619]: osdmap e139: 8 total, 8 up, 8 in 2024-03-04T16:07:25.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:25.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.355 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:25.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:25.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:25.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: pgmap v182: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 16645/177813 objects misplaced (9.361%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:07:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[29779]: osdmap e139: 8 total, 8 up, 8 in 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: pgmap v182: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 16645/177813 objects misplaced (9.361%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:07:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:25 smithi038 ceph-mon[38113]: osdmap e139: 8 total, 8 up, 8 in 2024-03-04T16:07:25.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:25.628 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:25.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.773 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:25.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:25.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:25.905 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:25.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:26.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:26.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:26.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:26.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:26.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:26.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:26.340 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:26.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:26.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:07:26.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:07:26.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:07:26.388 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:07:26.388 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:07:26.389 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:07:26.389 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:07:26.389 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:26 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:07:26.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:26.466 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:26.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:26.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:26.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:26 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:07:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:26 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:07:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:26 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:07:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:26 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:07:26.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:26.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:07:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:26.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:07:26.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:26.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:26.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:26.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:27.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:27.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:27.129 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:27.175 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:07:27.175 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:07:27.175 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:07:27.175 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:07:27.176 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:07:27.177 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:07:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:27.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:27.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:27.312 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:27.414 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:27.444 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[29779]: pgmap v184: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 768 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 563 objects/s recovering 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4183847830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[38113]: pgmap v184: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 768 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 563 objects/s recovering 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:07:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:07:27.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:27 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4183847830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:27.504 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:27 smithi176 ceph-mon[33619]: pgmap v184: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 768 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 563 objects/s recovering 2024-03-04T16:07:27.504 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:27 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:07:27.504 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:27 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:07:27.504 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:27 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4183847830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:27.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:27.575 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:27.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.722 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:27.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:27.883 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:27.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:27.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:28.030 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:28.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:28.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.177 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:28.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:28.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:28.352 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:28.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.367 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:28.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:28.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:28.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:28.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.667 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:28.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:28.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:28.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:28.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:28.931 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:28.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:29.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.067 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:29.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:29.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:29.210 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: pgmap v185: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 766 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 562 objects/s recovering 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:07:29.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: pgmap v185: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 766 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 562 objects/s recovering 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:07:29.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:28 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:07:29.262 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: pgmap v185: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 766 op/s; 16645/177813 objects misplaced (9.361%); 35 MiB/s, 562 objects/s recovering 2024-03-04T16:07:29.262 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.262 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:29.263 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:07:29.263 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:07:29.263 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:07:29.263 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:28 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:07:29.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:29.337 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:29.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:29.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:29.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:29.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:29.622 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:29.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:29.759 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:07:29.759 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:07:29.763 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:29.865 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:29.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:29.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:29.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:30.039 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:30.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:30.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:07:30.337 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:30.447 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:30.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:30.592 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:30.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:30.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:30.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.758 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:30.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:30.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:30.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:07:30.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:30.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:31.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:31.053 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:31.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.198 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:31.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:31.253 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:31.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.356 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:31.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:31.399 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:31 smithi176 ceph-mon[33619]: pgmap v186: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 12311/166230 objects misplaced (7.406%); 53 MiB/s, 844 objects/s recovering 2024-03-04T16:07:31.430 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:31 smithi038 ceph-mon[38113]: pgmap v186: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 12311/166230 objects misplaced (7.406%); 53 MiB/s, 844 objects/s recovering 2024-03-04T16:07:31.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:31 smithi038 ceph-mon[29779]: pgmap v186: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 12311/166230 objects misplaced (7.406%); 53 MiB/s, 844 objects/s recovering 2024-03-04T16:07:31.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:31.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:31.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:31.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:31.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:31.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:31.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:31.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:31.938 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:31.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:32.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:32.069 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:32.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:32.212 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:32.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:07:32.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:32 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:07:32.315 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:32.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:32.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:32.474 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:32.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:32 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:07:32.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:32 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:07:32.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:32 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:07:32.527 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:32 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:07:32.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:32.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:32.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:32.756 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:32.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:32.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:32.889 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:32.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:32.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:32.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:33.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:33.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:33.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:33.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:33.282 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:33.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:33 smithi176 ceph-mon[33619]: pgmap v187: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 482 op/s; 12311/166230 objects misplaced (7.406%); 26 MiB/s, 421 objects/s recovering 2024-03-04T16:07:33.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.419 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:33.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:33 smithi038 ceph-mon[29779]: pgmap v187: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 482 op/s; 12311/166230 objects misplaced (7.406%); 26 MiB/s, 421 objects/s recovering 2024-03-04T16:07:33.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:33 smithi038 ceph-mon[38113]: pgmap v187: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 482 op/s; 12311/166230 objects misplaced (7.406%); 26 MiB/s, 421 objects/s recovering 2024-03-04T16:07:33.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:33.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.557 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:33.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:33.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:33.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:33.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:33.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:33.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:33.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:33.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:34.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:34.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:34.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:34.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.237 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:34.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:34.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:34.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:34 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:07:34.276 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:34 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:07:34.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:34.379 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:34.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:34.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:34.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:07:34.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:07:34.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:34.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:07:34.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:34 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:07:34.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:34.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.673 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:34.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:34.774 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:34.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:34.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:34.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:35.056 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:07:35.056 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:07:35.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:35.207 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T16:07:35.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:35.249 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:35.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.377 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:35.444 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:35 smithi038 ceph-mon[29779]: pgmap v188: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 542 objects/s recovering 2024-03-04T16:07:35.444 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:07:35.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:35 smithi038 ceph-mon[38113]: pgmap v188: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 542 objects/s recovering 2024-03-04T16:07:35.444 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:07:35.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:35.502 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:35.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:35 smithi176 ceph-mon[33619]: pgmap v188: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 542 objects/s recovering 2024-03-04T16:07:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:07:35.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:35.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:07:35.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:35.758 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:35.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:35.878 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:35.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:35.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:35.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:35.965 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:35.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:36.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:36.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:36.148 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:36.250 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:36.275 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:07:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:07:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: 7.a deep-scrub starts 2024-03-04T16:07:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: pgmap v190: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 541 objects/s recovering 2024-03-04T16:07:36.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: 7.b scrub starts 2024-03-04T16:07:36.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:07:36.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:07:36.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:36 smithi176 ceph-mon[33619]: 7.b scrub ok 2024-03-04T16:07:36.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:36.403 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: 7.a deep-scrub starts 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: pgmap v190: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 541 objects/s recovering 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: 7.b scrub starts 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:07:36.446 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[29779]: 7.b scrub ok 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1281840922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: osdmap e140: 8 total, 8 up, 8 in 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: 7.a deep-scrub starts 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: pgmap v190: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 858 op/s; 9250/152070 objects misplaced (6.083%); 34 MiB/s, 541 objects/s recovering 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: 7.b scrub starts 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:07:36.447 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:36 smithi038 ceph-mon[38113]: 7.b scrub ok 2024-03-04T16:07:36.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 1 1 2024-03-04T16:07:36.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:36.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:36.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:36.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:36.778 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:36.801 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:36.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:36.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:36.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:37.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:37.054 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:37.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:37.181 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:37.198 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:07:37.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:37.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:07:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:07:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:07:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:07:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:07:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:07:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 6.18 deep-scrub starts 2024-03-04T16:07:37.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:37 smithi176 ceph-mon[33619]: 6.18 deep-scrub ok 2024-03-04T16:07:37.413 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:37.441 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:07:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:07:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:07:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 6.18 deep-scrub starts 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[29779]: 6.18 deep-scrub ok 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: osdmap e141: 8 total, 8 up, 8 in 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 6.18 deep-scrub starts 2024-03-04T16:07:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:37 smithi038 ceph-mon[38113]: 6.18 deep-scrub ok 2024-03-04T16:07:37.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:37.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:37.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:37.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:37.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:37.867 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:37.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:37.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:38.005 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:38.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:38.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:38.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.278 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:38.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:38.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:38.416 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: osdmap e142: 8 total, 8 up, 8 in 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: 7.6 starting backfill to osd.0 from (117'6186,137'11043] MIN to 139'11736 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[29779]: pgmap v193: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 9250/152070 objects misplaced (6.083%); 21 MiB/s, 340 objects/s recovering 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: osdmap e142: 8 total, 8 up, 8 in 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: 7.6 starting backfill to osd.0 from (117'6186,137'11043] MIN to 139'11736 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:38 smithi038 ceph-mon[38113]: pgmap v193: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 9250/152070 objects misplaced (6.083%); 21 MiB/s, 340 objects/s recovering 2024-03-04T16:07:38.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:38.556 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:38.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3194150978' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: osdmap e142: 8 total, 8 up, 8 in 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: 7.6 starting backfill to osd.0 from (117'6186,137'11043] MIN to 139'11736 2024-03-04T16:07:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:38.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:38 smithi176 ceph-mon[33619]: pgmap v193: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 9250/152070 objects misplaced (6.083%); 21 MiB/s, 340 objects/s recovering 2024-03-04T16:07:38.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.691 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:38.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:38.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:38.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:38.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:38.967 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:38.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:39.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:39.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:39.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:39.242 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:39.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:39.372 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:39.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[29779]: osdmap e143: 8 total, 8 up, 8 in 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:39 smithi038 ceph-mon[38113]: osdmap e143: 8 total, 8 up, 8 in 2024-03-04T16:07:39.499 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:39.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:39.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:39 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:39 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:39 smithi176 ceph-mon[33619]: osdmap e143: 8 total, 8 up, 8 in 2024-03-04T16:07:39.634 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:39.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:39.764 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:39.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:39.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:39.906 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:39.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:40.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:40.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:40.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:40.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:40.231 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:40.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:40.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[29779]: osdmap e144: 8 total, 8 up, 8 in 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[29779]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 679 op/s; 2996/143934 objects misplaced (2.082%); 17 MiB/s, 289 objects/s recovering 2024-03-04T16:07:40.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[38113]: osdmap e144: 8 total, 8 up, 8 in 2024-03-04T16:07:40.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:40.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:40.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:40.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:40 smithi038 ceph-mon[38113]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 679 op/s; 2996/143934 objects misplaced (2.082%); 17 MiB/s, 289 objects/s recovering 2024-03-04T16:07:40.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.449 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:40.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:40.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:40 smithi176 ceph-mon[33619]: osdmap e144: 8 total, 8 up, 8 in 2024-03-04T16:07:40.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:40 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:07:40.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:40 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:07:40.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:40 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:40.516 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:40 smithi176 ceph-mon[33619]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 679 op/s; 2996/143934 objects misplaced (2.082%); 17 MiB/s, 289 objects/s recovering 2024-03-04T16:07:40.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:40.591 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:40.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:40.721 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:40.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:40.848 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:40.922 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:07:40.950 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:40.955 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:07:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:40.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:40.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:40.987 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:07:41.021 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:07:41.052 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:07:41.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:41.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:07:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.117 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:41.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:41.118 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:07:41.162 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:07:41.167 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:07:41.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:41.248 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:41.336 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:07:41.337 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:07:41.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:41.374 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:41.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.502 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:41.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:41.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:41 smithi038 ceph-mon[29779]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:07:41.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4245547304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:41.595 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:41 smithi038 ceph-mon[38113]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:07:41.595 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4245547304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:41.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:41 smithi176 ceph-mon[33619]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:07:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4245547304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:41.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:41.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:41.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:07:41.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:41.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:41.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:41.906 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:41.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:42.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:42.055 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:42.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:42.188 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:42.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:42.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:42.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:07:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:42.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:42.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.465 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:42.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:42.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:42 smithi176 ceph-mon[33619]: pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 523 op/s; 2996/143934 objects misplaced (2.082%); 13 MiB/s, 223 objects/s recovering 2024-03-04T16:07:42.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:42.592 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:42.619 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:42.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:42.721 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:42.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:42 smithi038 ceph-mon[29779]: pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 523 op/s; 2996/143934 objects misplaced (2.082%); 13 MiB/s, 223 objects/s recovering 2024-03-04T16:07:42.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:42 smithi038 ceph-mon[38113]: pgmap v197: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+undersized+remapped, 1 active+remapped+backfilling, 145 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 987 B/s rd, 0 B/s wr, 523 op/s; 2996/143934 objects misplaced (2.082%); 13 MiB/s, 223 objects/s recovering 2024-03-04T16:07:42.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:42.850 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:42.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:42.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:42.976 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:42.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:43.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:43.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:43.206 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:43.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:43.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:43 smithi038 ceph-mon[29779]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:07:43.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:43.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:43 smithi038 ceph-mon[38113]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:07:43.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:43.319 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:43.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:43.361 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:43.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.515 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:43.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:43 smithi176 ceph-mon[33619]: osdmap e145: 8 total, 8 up, 8 in 2024-03-04T16:07:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:43.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:43.624 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:43.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:43.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:43.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:43.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:43.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:43.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:43.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:44.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:44.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:44.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.198 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:44.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:44.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:44.306 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T16:07:44.327 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:44.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:44.433 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:44 smithi176 ceph-mon[33619]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:07:44.433 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:44 smithi176 ceph-mon[33619]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 717 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:44.433 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1289821463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.433 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:44 smithi176 ceph-mon[33619]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[29779]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[29779]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 717 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1289821463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[29779]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[38113]: osdmap e146: 8 total, 8 up, 8 in 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[38113]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 996 B/s rd, 0 B/s wr, 717 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1289821463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:44 smithi038 ceph-mon[38113]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:07:44.459 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:44.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:44.590 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:07:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:07:44.590 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:07:44.591 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.601 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:44.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:44.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.745 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:44.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:44.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:07:44.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:44.888 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:44.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:44.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.013 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:45.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:45.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:45.163 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:45.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:45.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:45.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:45.434 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:45.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:45.580 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:45 smithi176 ceph-mon[33619]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:07:45.580 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:45 smithi176 ceph-mon[33619]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:07:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.581 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:45.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:45.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:07:45.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:45 smithi038 ceph-mon[29779]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:07:45.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:45 smithi038 ceph-mon[29779]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:07:45.656 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:45 smithi038 ceph-mon[38113]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:07:45.656 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:45 smithi038 ceph-mon[38113]: osdmap e147: 8 total, 8 up, 8 in 2024-03-04T16:07:45.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:45.720 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:45.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:45.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:45.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:45.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:45.947 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:07:45.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:46.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 2 1 2024-03-04T16:07:46.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:46.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:46.184 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:46.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:46.285 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.315 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:46.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:46.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:46.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:46.554 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:46.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:46 smithi176 ceph-mon[33619]: pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 614 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-04T16:07:46.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:46.725 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:46 smithi038 ceph-mon[29779]: pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 614 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-04T16:07:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:46 smithi038 ceph-mon[38113]: pgmap v202: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 614 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-04T16:07:46.789 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:46.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:46.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:46.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:46.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.033 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:47.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:47.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:47.165 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:47.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:47.314 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:47.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:47.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/636654307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:47 smithi176 ceph-mon[33619]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.362 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:07:47.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/636654307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:47 smithi038 ceph-mon[29779]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/636654307' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.362 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:47 smithi038 ceph-mon[38113]: from='client.24910 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:07:47.417 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:47.444 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:47.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:47.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:47.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:47.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:47.795 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:47.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:47.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:47.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:47.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:48.005 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:48.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83618 7.5 7:a4d43b88:::benchmark_data_smithi038_77863_object83617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.279983+0000", 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 127.769335957, 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12038834, 2024-03-04T16:07:48.053 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83618 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279983+0000", 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279983+0000", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279992+0000", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8089999999999999e-06 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280022+0000", 2024-03-04T16:07:48.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0805000000000001e-05 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280035+0000", 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2061e-05 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280043+0000", 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7450000000000001e-06 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280107+0000", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3602999999999994e-05 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280998+0000", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00089115899999999998 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,2", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.281209+0000", 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000210977 2024-03-04T16:07:48.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.284152+0000", 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029433020000000001 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.287302+0000", 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031492600000000001 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400250+0000", 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11294889900000001 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400315+0000", 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4488999999999999e-05 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400371+0000", 2024-03-04T16:07:48.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6230999999999997e-05 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159706 7.5 7:aa279c05:::benchmark_data_smithi038_77863_object41438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.637644+0000", 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 47.411675002999999, 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132516893, 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.060 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159706 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637644+0000", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637644+0000", 2024-03-04T16:07:48.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637646+0000", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2469999999999998e-06 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637652+0000", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0809999999999999e-06 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637664+0000", 2024-03-04T16:07:48.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1958e-05 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637672+0000", 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8909999999999995e-06 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637728+0000", 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5760000000000001e-05 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637834+0000", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105753 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,2", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637917+0000", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3796999999999997e-05 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.640606+0000", 2024-03-04T16:07:48.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026884719999999999 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643283+0000", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026772779999999999 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.770047+0000", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12676353700000001 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.770110+0000", 2024-03-04T16:07:48.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3159999999999998e-05 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.770161+0000", 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0958999999999998e-05 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163736 7.6 7:6ebeca7a:::benchmark_data_smithi038_77863_object45468:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.199868+0000", 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.849451033000001, 2024-03-04T16:07:48.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.160784252, 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163736 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.067 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199868+0000", 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199868+0000", 2024-03-04T16:07:48.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199870+0000", 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2060000000000001e-06 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199876+0000", 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.387e-06 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199887+0000", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1090999999999999e-05 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199896+0000", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6789999999999994e-06 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204721+0000", 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048244179999999996 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204730+0000", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0780000000000002e-06 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.214754+0000", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0100248359999999 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.214768+0000", 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.329e-05 2024-03-04T16:07:48.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.358623+0000", 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.143855405 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360652+0000", 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020288620000000002 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.072 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163740 7.6 7:601e938b:::benchmark_data_smithi038_77863_object45472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.199911+0000", 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 42.849408298, 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.161127998, 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163740 2024-03-04T16:07:48.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199911+0000", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199911+0000", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199911+0000", 2024-03-04T16:07:48.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2200000000000003e-07 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199914+0000", 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6249999999999999e-06 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199920+0000", 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7939999999999999e-06 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.199925+0000", 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5269999999999998e-06 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204754+0000", 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0048286049999999997 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204761+0000", 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8680000000000001e-06 2024-03-04T16:07:48.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.214843+0000", 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0100821870000001 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.214852+0000", 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2429999999999994e-06 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360925+0000", 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.146072859 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.361039+0000", 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011356799999999999 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.080 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178257 7.6 7:6b287665:::benchmark_data_smithi038_77863_object59989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.360382+0000", 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.688936722000001, 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.034382291, 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.081 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178257 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360382+0000", 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360382+0000", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360384+0000", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5370000000000001e-06 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360389+0000", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6099999999999997e-06 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360402+0000", 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2840000000000001e-05 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360410+0000", 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6420000000000007e-06 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360467+0000", 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7760000000000003e-05 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360547+0000", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9950000000000005e-05 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3,5", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.360619+0000", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2020999999999994e-05 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.362689+0000", 2024-03-04T16:07:48.085 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020699920000000001 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.363000+0000", 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031081799999999998 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.366881+0000", 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003881224 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.366894+0000", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2741999999999999e-05 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.394687+0000", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0277928919999999 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.394764+0000", 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7262999999999995e-05 2024-03-04T16:07:48.087 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178259 7.6 7:6de485a8:::benchmark_data_smithi038_77863_object59991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.365719+0000", 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.683599924999999, 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.039024927, 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.088 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178259 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365719+0000", 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365719+0000", 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365720+0000", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.598e-06 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365724+0000", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6200000000000001e-06 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365730+0000", 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0889999999999996e-06 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.365735+0000", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0010000000000001e-06 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.366935+0000", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011996000000000001 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.366944+0000", 2024-03-04T16:07:48.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5589999999999994e-06 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.395206+0000", 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.028261715 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.395334+0000", 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012788900000000001 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.395485+0000", 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015077299999999999 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400590+0000", 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051057639999999996 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400713+0000", 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122965 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404645+0000", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039316869999999997 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404700+0000", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4577000000000003e-05 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404744+0000", 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.409e-05 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.094 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178262 7.6 7:615b7522:::benchmark_data_smithi038_77863_object59994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.371254+0000", 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.678064889000002, 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.040067109, 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178262 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.095 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371254+0000", 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371254+0000", 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371256+0000", 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.64e-06 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371260+0000", 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7570000000000004e-06 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371270+0000", 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6059999999999992e-06 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371278+0000", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8199999999999997e-06 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371346+0000", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8138000000000004e-05 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.371354+0000", 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1110000000000008e-06 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.398386+0000", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0270324829999999 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.398513+0000", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012619700000000001 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.398650+0000", 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013738499999999999 2024-03-04T16:07:48.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400627+0000", 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001977342 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.406726+0000", 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060990860000000001 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411240+0000", 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045139539999999997 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411272+0000", 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2051000000000002e-05 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411321+0000", 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8538999999999997e-05 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178294 7.6 7:65824f12:::benchmark_data_smithi038_77863_object60026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.452798+0000", 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.596520388999998, 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96255564599999999, 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178294 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452798+0000", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452798+0000", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452801+0000", 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.311e-06 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452807+0000", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3149999999999997e-06 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452824+0000", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7164999999999999e-05 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452833+0000", 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7050000000000005e-06 2024-03-04T16:07:48.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452904+0000", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0693999999999998e-05 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.452913+0000", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5480000000000007e-06 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400751+0000", 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94783783300000002 2024-03-04T16:07:48.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400871+0000", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120006 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.400978+0000", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010682499999999999 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404781+0000", 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038036379999999998 2024-03-04T16:07:48.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411126+0000", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063441390000000004 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415242+0000", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041160830000000004 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415318+0000", 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5850000000000001e-05 2024-03-04T16:07:48.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415354+0000", 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6534e-05 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178298 7.6 7:66d69bcf:::benchmark_data_smithi038_77863_object60030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.453307+0000", 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.596011405999999, 2024-03-04T16:07:48.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96220282400000001, 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178298 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453307+0000", 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453307+0000", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453309+0000", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.333e-06 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453313+0000", 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.639e-06 2024-03-04T16:07:48.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453322+0000", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7900000000000005e-06 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453328+0000", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6180000000000003e-06 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453369+0000", 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1702000000000002e-05 2024-03-04T16:07:48.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.453375+0000", 2024-03-04T16:07:48.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2320000000000001e-06 2024-03-04T16:07:48.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:48.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.402685+0000", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94931058899999998 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.402801+0000", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115344 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.402940+0000", 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139128 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.406791+0000", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038514790000000001 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411133+0000", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043418830000000004 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415408+0000", 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042745969999999998 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415463+0000", 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5170000000000002e-05 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415510+0000", 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7320000000000001e-05 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178327 7.6 7:6ff23d60:::benchmark_data_smithi038_77863_object60059:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.479687+0000", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.569632151, 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93601931000000005, 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178327 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479687+0000", 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479687+0000", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479689+0000", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.294e-06 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479695+0000", 2024-03-04T16:07:48.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9839999999999998e-06 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479706+0000", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1531e-05 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479714+0000", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5939999999999996e-06 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479770+0000", 2024-03-04T16:07:48.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5457000000000001e-05 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.479778+0000", 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6000000000000007e-06 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404829+0000", 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92505127200000004 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.404943+0000", 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011332099999999999 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.405065+0000", 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012226199999999999 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411173+0000", 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006107726 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411213+0000", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0250999999999998e-05 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415643+0000", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044300809999999998 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415673+0000", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0173000000000001e-05 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415706+0000", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2764e-05 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178331 7.6 7:6376a98a:::benchmark_data_smithi038_77863_object60063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.483149+0000", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.566169509000002, 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94496714199999998, 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.123 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178331 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483149+0000", 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483149+0000", 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483151+0000", 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.669e-06 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483155+0000", 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4689999999999999e-06 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483165+0000", 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6020000000000006e-06 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483171+0000", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7389999999999996e-06 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483216+0000", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5563999999999999e-05 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.483223+0000", 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1800000000000001e-06 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.406836+0000", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92361298999999997 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.406980+0000", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014464100000000001 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.407110+0000", 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012959399999999999 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411205+0000", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040953070000000003 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411217+0000", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1545999999999999e-05 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428036+0000", 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016818976999999999 2024-03-04T16:07:48.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428084+0000", 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8361e-05 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428116+0000", 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2502999999999997e-05 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.129 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178334 7.6 7:6b0d289e:::benchmark_data_smithi038_77863_object60066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.486555+0000", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.562764313999999, 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94192593800000002, 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178334 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486555+0000", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486555+0000", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486556+0000", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.218e-06 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486560+0000", 2024-03-04T16:07:48.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1949999999999996e-06 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486567+0000", 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1890000000000001e-06 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486572+0000", 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2580000000000003e-06 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486614+0000", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1551999999999998e-05 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.486621+0000", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6930000000000001e-06 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.409100+0000", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92247977699999995 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.409219+0000", 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011862900000000001 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.409353+0000", 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013357000000000001 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411371+0000", 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020188070000000001 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415182+0000", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038105499999999998 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428398+0000", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013215806 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428446+0000", 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8381000000000003e-05 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428480+0000", 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4313000000000002e-05 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178344 7.6 7:613f919e:::benchmark_data_smithi038_77863_object60076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.494993+0000", 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.554326100000001, 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93358513499999995, 2024-03-04T16:07:48.136 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178344 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.494993+0000", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.494993+0000", 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.494994+0000", 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1069999999999999e-06 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.494997+0000", 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2650000000000001e-06 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.138 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.495008+0000", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1e-05 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.495015+0000", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2450000000000001e-06 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.495068+0000", 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2889999999999997e-05 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.495076+0000", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6140000000000003e-06 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411405+0000", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.916328963 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411472+0000", 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6713999999999999e-05 2024-03-04T16:07:48.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.411534+0000", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2456e-05 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415522+0000", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039881359999999998 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415553+0000", 2024-03-04T16:07:48.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0670000000000003e-05 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428525+0000", 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012972529999999999 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428555+0000", 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9498000000000001e-05 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428578+0000", 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3047000000000001e-05 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178357 7.6 7:6c2d22e2:::benchmark_data_smithi038_77863_object60089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.511468+0000", 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.53785061, 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.91719197600000002, 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178357 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511468+0000", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511468+0000", 2024-03-04T16:07:48.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511470+0000", 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0480000000000001e-06 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511476+0000", 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.643e-06 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511487+0000", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0845e-05 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511495+0000", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0709999999999995e-06 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511551+0000", 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6462000000000003e-05 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.511557+0000", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.913e-06 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.413168+0000", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90161108400000001 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.413283+0000", 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114929 2024-03-04T16:07:48.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.413419+0000", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013544699999999999 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415597+0000", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002178546 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415613+0000", 2024-03-04T16:07:48.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5953000000000001e-05 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428613+0000", 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012999638000000001 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428640+0000", 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7069000000000001e-05 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.149 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428660+0000", 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0327999999999999e-05 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178362 7.6 7:6cdebd95:::benchmark_data_smithi038_77863_object60094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.515546+0000", 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.533773092000001, 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92643682400000005, 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.150 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178362 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515546+0000", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515546+0000", 2024-03-04T16:07:48.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515548+0000", 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.176e-06 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515555+0000", 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7750000000000004e-06 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515566+0000", 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1269e-05 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515574+0000", 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9019999999999999e-06 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515630+0000", 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6007000000000001e-05 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.515636+0000", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.387e-06 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415756+0000", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90011988600000004 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415894+0000", 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013753499999999999 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.415978+0000", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4766999999999997e-05 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.421552+0000", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0055734959999999998 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.423853+0000", 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023005870000000002 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.441876+0000", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018023170000000002 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.441932+0000", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5964999999999997e-05 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.441983+0000", 2024-03-04T16:07:48.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0902000000000002e-05 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178384 7.6 7:65074acd:::benchmark_data_smithi038_77863_object60116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.542056+0000", 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.507262916999998, 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90007947700000002, 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.157 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178384 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542056+0000", 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542056+0000", 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542058+0000", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6000000000000001e-06 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542064+0000", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4890000000000003e-06 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542079+0000", 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5184999999999999e-05 2024-03-04T16:07:48.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542124+0000", 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4391999999999998e-05 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542179+0000", 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5300999999999998e-05 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.542187+0000", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5469999999999997e-06 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.418404+0000", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87621615799999997 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.418511+0000", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107831 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.418624+0000", 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112317 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.421613+0000", 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029893469999999998 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.427981+0000", 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006368065 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442040+0000", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014058777999999999 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442094+0000", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4289999999999997e-05 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442135+0000", 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1177000000000002e-05 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.163 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178401 7.6 7:68443ebe:::benchmark_data_smithi038_77863_object60133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.566113+0000", 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.483206183, 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87618569800000001, 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178401 2024-03-04T16:07:48.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566113+0000", 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566113+0000", 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566115+0000", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0650000000000001e-06 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566121+0000", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4690000000000002e-06 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566135+0000", 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3788000000000001e-05 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566142+0000", 2024-03-04T16:07:48.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5980000000000002e-06 2024-03-04T16:07:48.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566197+0000", 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5801000000000003e-05 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.566206+0000", 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8540000000000003e-06 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.421658+0000", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85545166800000005 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.421771+0000", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112893 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.421836+0000", 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5248999999999998e-05 2024-03-04T16:07:48.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428170+0000", 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0063338960000000003 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428186+0000", 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5758999999999999e-05 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442195+0000", 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014009054 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442248+0000", 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2970000000000003e-05 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442298+0000", 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0633999999999997e-05 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178415 7.6 7:6c37eea3:::benchmark_data_smithi038_77863_object60147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.586578+0000", 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.462740457999999, 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85591669699999995, 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.172 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178415 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586578+0000", 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586578+0000", 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586581+0000", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1739999999999999e-06 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586586+0000", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0900000000000004e-06 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586596+0000", 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0553e-05 2024-03-04T16:07:48.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586603+0000", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4590000000000003e-06 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586673+0000", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0321000000000007e-05 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.586681+0000", 2024-03-04T16:07:48.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0569999999999993e-06 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.423885+0000", 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.83720409399999995 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.423958+0000", 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3201000000000006e-05 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.424036+0000", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8152000000000007e-05 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428192+0000", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041556600000000003 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428240+0000", 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7936999999999999e-05 2024-03-04T16:07:48.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442409+0000", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014168505 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442457+0000", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8779999999999997e-05 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442495+0000", 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7713999999999998e-05 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.178 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178422 7.6 7:65db45c6:::benchmark_data_smithi038_77863_object60154:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.599765+0000", 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.449553625, 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84289037899999997, 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178422 2024-03-04T16:07:48.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599765+0000", 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599765+0000", 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.180 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599767+0000", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8899999999999999e-06 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599772+0000", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3129999999999998e-06 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599781+0000", 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0049999999999993e-06 2024-03-04T16:07:48.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599788+0000", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2210000000000002e-06 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599831+0000", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3634999999999998e-05 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.599837+0000", 2024-03-04T16:07:48.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9070000000000004e-06 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.425493+0000", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.82565539300000002 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.425627+0000", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013416799999999999 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.425696+0000", 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.957e-05 2024-03-04T16:07:48.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428490+0000", 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027940389999999999 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428692+0000", 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020167699999999999 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442573+0000", 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013880751 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442621+0000", 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.791e-05 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.442656+0000", 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4900000000000001e-05 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.185 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:178431 7.6 7:6d81943f:::benchmark_data_smithi038_77863_object60163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:23.630189+0000", 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.419130306, 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.82427093699999998, 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178431 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630189+0000", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630189+0000", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630191+0000", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0030000000000001e-06 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:07:48.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630196+0000", 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1279999999999999e-06 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630206+0000", 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0132000000000001e-05 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630217+0000", 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1236999999999999e-05 2024-03-04T16:07:48.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630270+0000", 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2575999999999999e-05 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:23.630282+0000", 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2866e-05 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:07:48.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428720+0000", 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.79843725600000004 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428789+0000", 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9665000000000005e-05 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.428855+0000", 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5990999999999999e-05 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.435541+0000", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0066852760000000004 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.435620+0000", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9617999999999996e-05 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.454363+0000", 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.018742734 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.454417+0000", 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3978999999999999e-05 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:24.454459+0000", 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2475999999999998e-05 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:07:48.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:07:48.193 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:07:48.194 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:48.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:48.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:48.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:48.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:48.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:48.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[29779]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:48.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[29779]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:07:48.549 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[29779]: pgmap v204: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 770 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:48.550 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:48.550 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[38113]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:48.550 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[38113]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:07:48.550 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:48 smithi038 ceph-mon[38113]: pgmap v204: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 770 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:48.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:48.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:48 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:07:48.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:48 smithi176 ceph-mon[33619]: from='client.24910 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:07:48.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:48 smithi176 ceph-mon[33619]: osdmap e148: 8 total, 8 up, 8 in 2024-03-04T16:07:48.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:48 smithi176 ceph-mon[33619]: pgmap v204: 148 pgs: 1 active+clean+scrubbing+deep, 1 peering, 146 active+clean; 2.7 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 770 op/s; 3.0 MiB/s, 48 objects/s recovering 2024-03-04T16:07:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.619 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:48.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:48.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:48.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:48.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:48.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:48.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:48.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:07:48.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:49.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:49.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:49.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:49.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:49.298 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:49.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:49.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:49.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:49.559 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:49.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:49.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:49.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.831 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:49.933 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:49.962 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:49.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:50.044 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:07:50.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:50.099 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:07:50.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:50.146 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:07:50.195 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:07:50.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:50.234 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:07:50.260 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:50.275 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:07:50.313 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:07:50.333 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:07:50.334 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:07:50.334 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:07:50.334 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:07:50.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:07:50.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:07:50.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:07:50.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:07:50.335 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:07:50.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:50.388 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:50.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:50.531 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:50.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:50.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:50 smithi038 ceph-mon[29779]: pgmap v205: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 594 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3206178837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:50 smithi038 ceph-mon[38113]: pgmap v205: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 594 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3206178837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:50.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:50.801 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:50 smithi176 ceph-mon[33619]: pgmap v205: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 840 B/s rd, 0 B/s wr, 594 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3206178837' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:07:50.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:50.943 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:50.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:51.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:51.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:51.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:51.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:51.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:51.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:51.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:51.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:51.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:07:51.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.459 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:51.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:51.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:51.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:51.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:51.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:51.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:51.877 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:51.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:51.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.019 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:52.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:52.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.158 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:52.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:52.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:52.289 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:52.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:52.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:52.437 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:52.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:52.570 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:52.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:52.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:52.710 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:52.812 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:52.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:52.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:52 smithi176 ceph-mon[33619]: pgmap v206: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:52 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:52.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:07:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:07:52.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:52.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:52.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[29779]: pgmap v206: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[38113]: pgmap v206: 148 pgs: 148 active+clean; 2.5 GiB data, 8.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:07:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:07:53.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:53.114 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:53.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:53.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:53.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:53.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:53.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:53.504 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:53.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:53.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:53.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:53.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:07:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:07:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:07:53.767 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:53.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:53.905 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:53.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:54.007 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:54.037 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:54.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:54.179 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:54.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:54.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:07:54.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:54.311 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:54.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:54.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.443 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:54.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:54.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:54.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:54.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:54.715 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:54.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:54.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:54.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:54.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:54.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:54.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:54 smithi038 ceph-mon[29779]: pgmap v207: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 938 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:54 smithi038 ceph-mon[38113]: pgmap v207: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 938 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.000 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:55.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:55.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:54 smithi176 ceph-mon[33619]: pgmap v207: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 938 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:55.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:55.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:55.275 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:55.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:55.405 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:55.460 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:55.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:55.554 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:55.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:55.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:55.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:55.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:55.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:55.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:55.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:55.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:55.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:56.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:56.077 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:56.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:56.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:56.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:56.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.350 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:56.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:56.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.481 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:56.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:56.526 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:07:56.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.644 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:56.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:56.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:56.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:56.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:56.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:56.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:56.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[29779]: pgmap v208: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:07:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:07:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[38113]: pgmap v208: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:07:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:56 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:07:57.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:57.080 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:56 smithi176 ceph-mon[33619]: pgmap v208: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 852 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:56 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:07:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:56 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:07:57.183 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:57.210 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:57.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:57.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:57.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:57.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:57.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:07:57.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:57.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:57.737 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:57.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:57.825 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T16:07:57.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:57.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:57.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:58.026 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:07:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.064 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:58.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:07:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:57 smithi176 ceph-mon[33619]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:07:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3071972820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:57 smithi176 ceph-mon[33619]: from='client.34922 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:07:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.220 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:58.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[29779]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3071972820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[29779]: from='client.34922 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[38113]: osdmap e149: 8 total, 8 up, 8 in 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3071972820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:57 smithi038 ceph-mon[38113]: from='client.34922 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:07:58.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:58.354 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:58.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:58.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:58.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:58.647 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:58.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:58.809 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:58.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[38113]: pgmap v210: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:58.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[38113]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:07:58.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[38113]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:07:58.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[29779]: pgmap v210: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:58.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[29779]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:07:58.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:07:58 smithi038 ceph-mon[29779]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:07:58.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:07:58.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:58.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:07:59.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:59.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:59.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:58 smithi176 ceph-mon[33619]: pgmap v210: 148 pgs: 148 active+clean; 2.2 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 828 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:07:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:58 smithi176 ceph-mon[33619]: from='client.34922 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:07:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:07:58 smithi176 ceph-mon[33619]: osdmap e150: 8 total, 8 up, 8 in 2024-03-04T16:07:59.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:07:59.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:07:59.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:07:59.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:07:59.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:07:59.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:07:59.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:59.608 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:07:59.608 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:07:59.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.620 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:59.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:59.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:07:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.754 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:07:59.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:07:59.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:07:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:07:59.891 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:07:59.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:07:59.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.032 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:00.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:00.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:00.162 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:00.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:00.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.296 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:00.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:00.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.441 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:00.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:00.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:00.583 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:00.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:00.720 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:00.822 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:00.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 3 1 2024-03-04T16:08:00.847 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:00.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:00.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:00.979 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:00.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:01.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:01.108 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:00 smithi176 ceph-mon[33619]: pgmap v212: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:01.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:01.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:00 smithi038 ceph-mon[38113]: pgmap v212: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:01.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:00 smithi038 ceph-mon[29779]: pgmap v212: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:01.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:01.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:01.405 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:01.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:08:01.474 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:01.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:01.641 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:01.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.807 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:01.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:01.860 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:08:01.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:01.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:01.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:02.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:02.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:02.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:02.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:02.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:02.216 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:02.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:02.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:02.472 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:02.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:02.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:02.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:02.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:02.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:02.845 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:02.864 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:02.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:02.943 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:02.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:03.027 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s 2024-03-04T16:08:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:02 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s 2024-03-04T16:08:03.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: pgmap v213: 148 pgs: 148 active+clean; 1.9 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 616 op/s 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:03.110 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2231028191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: osdmap e151: 8 total, 8 up, 8 in 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:08:03.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:02 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:08:03.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.159 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:03.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:03.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:03.300 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:03.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.446 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:03.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:03.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:03.553 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.601 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:03.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:03.759 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:03.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:03.904 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:03.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:04.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:04.056 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[29779]: 7.9 scrub starts 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[29779]: 7.a scrub starts 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:08:04.094 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[38113]: 7.9 scrub starts 2024-03-04T16:08:04.095 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:03 smithi038 ceph-mon[38113]: 7.a scrub starts 2024-03-04T16:08:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:03 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:08:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:03 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:08:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:03 smithi176 ceph-mon[33619]: 7.9 scrub starts 2024-03-04T16:08:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:03 smithi176 ceph-mon[33619]: 7.a scrub starts 2024-03-04T16:08:04.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:04.192 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:04.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.321 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:04.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:04.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:04.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:04.554 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:04.583 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:04.621 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:08:04.674 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:08:04.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:04.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:04.725 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:08:04.768 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:08:04.810 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:08:04.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:04.850 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:08:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:04.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:04.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:04.892 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:08:04.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:08:04.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:08:04.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: pgmap v215: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 7.9 scrub ok 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 7.a scrub ok 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: 7.3 scrub starts 2024-03-04T16:08:04.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3447635876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: pgmap v215: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 7.9 scrub ok 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 7.a scrub ok 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: 7.3 scrub starts 2024-03-04T16:08:04.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:04 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3447635876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:04.940 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:08:04.940 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:08:04.940 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:08:04.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:08:04.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:08:04.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:08:04.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:08:04.941 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:08:04.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:05.011 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:05.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:05.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:05.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:05.308 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:05.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: pgmap v215: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:08:05.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 7.9 scrub ok 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 7.a scrub ok 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: 7.3 scrub starts 2024-03-04T16:08:05.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:04 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3447635876' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:05.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:05.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:05.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:05.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:05.717 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:05.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:05.856 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:05.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:05.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:05.991 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:05.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:06.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:06.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:06.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:06.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:06.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.438 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:06.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[29779]: 7.3 scrub ok 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[38113]: 7.3 scrub ok 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:08:06.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:06 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:08:06.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:06.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:06 smithi176 ceph-mon[33619]: 7.3 scrub ok 2024-03-04T16:08:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:06 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:08:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:06 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:08:06.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:06.718 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:06.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:06.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:06.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:06.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:06.999 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:07.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:07.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:07.134 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:07.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:07.263 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:07.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:07.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:07.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: pgmap v216: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:08:07.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:08:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:08:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:08:07.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:07.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: pgmap v216: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:08:07.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:08:07.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:07.520 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: pgmap v216: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:08:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:08:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:08:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:07.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:07.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:07.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:07.808 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:07.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:07.955 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:07.991 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:08:07.991 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:08:07.991 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:08:07.991 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:08:07.991 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:08:08.057 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:08.089 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:08.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:08.224 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:08.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:08.357 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:08.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:08.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: pgmap v218: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 644 op/s 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:08:08.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: pgmap v218: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 644 op/s 2024-03-04T16:08:08.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:08:08.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:08:08.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:08:08.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:08 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:08:08.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:08.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:08.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:08:08.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: osdmap e152: 8 total, 8 up, 8 in 2024-03-04T16:08:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: pgmap v218: 148 pgs: 148 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 644 op/s 2024-03-04T16:08:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:08:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:08:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:08:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:08 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:08:08.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:08.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.773 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:08.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:08.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:08.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:08.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:09.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.042 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:09.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:09.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:09.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:09.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:09.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:08:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:09.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:09 smithi176 ceph-mon[33619]: 7.2 scrub starts 2024-03-04T16:08:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:09 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:08:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:09 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:08:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:09 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:08:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:09 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:08:09.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:09.468 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:09.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[29779]: 7.2 scrub starts 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:08:09.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[38113]: 7.2 scrub starts 2024-03-04T16:08:09.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:08:09.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:08:09.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:08:09.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:09 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:08:09.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.626 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:09.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:09.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:09.876 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:09.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:09.904 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:09.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:10.006 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:10.029 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:10.131 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.171 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:10.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:10.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:10.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:10.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.485 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:10.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: 7.2 scrub ok 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: pgmap v219: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: 7.2 scrub ok 2024-03-04T16:08:10.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: pgmap v219: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:10.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:08:10.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:08:10.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:08:10.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:10 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:08:10.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: 7.2 scrub ok 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: pgmap v219: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:08:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:10 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:08:10.615 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:10.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:10.743 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:10.845 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:10.876 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:10.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:10.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:11.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:11.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:11.161 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:11.217 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:11.217 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:11.231 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T16:08:11.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.312 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:11.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:11.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:08:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:08:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:08:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:08:11.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:08:11.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:08:11.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:08:11.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:08:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:08:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:08:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:08:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:08:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:08:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:08:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:08:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:08:11.694 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:11.738 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:11.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:11.887 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:11.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:11.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:12.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:12.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.162 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:12.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:12.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:12.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:12.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:12.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:12.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: 7.d scrub starts 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: 7.d scrub ok 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: pgmap v221: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: 7.8 scrub starts 2024-03-04T16:08:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: 7.d scrub starts 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: 7.d scrub ok 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: pgmap v221: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: 7.8 scrub starts 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:08:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:12 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:08:12.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:12.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:12.572 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: 7.d scrub starts 2024-03-04T16:08:12.572 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: 7.d scrub ok 2024-03-04T16:08:12.572 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2624040128' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:08:12.572 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: osdmap e153: 8 total, 8 up, 8 in 2024-03-04T16:08:12.572 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: pgmap v221: 148 pgs: 148 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 611 op/s 2024-03-04T16:08:12.573 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: 7.8 scrub starts 2024-03-04T16:08:12.573 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:08:12.573 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:08:12.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 4 1 2024-03-04T16:08:12.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:12.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:12.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:12.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:12.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:12.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:12.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:12.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:12.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:13.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:13.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:13.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:13.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:13.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:13.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:13.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.549 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:13.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 7.8 scrub ok 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:08:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:08:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:08:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:08:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:08:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:13 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:08:13.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:13.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 7.8 scrub ok 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:08:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 7.8 scrub ok 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:08:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:08:13.745 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:13.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.824 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:13.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:13.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:13.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:13.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:14.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:14.110 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:14.212 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:14.240 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:14.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:14.368 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:14.469 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[29779]: pgmap v222: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:14.469 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:08:14.470 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:08:14.470 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[38113]: pgmap v222: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:14.470 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:08:14.470 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:14 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:08:14.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.510 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:14.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:14 smithi176 ceph-mon[33619]: pgmap v222: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:14 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:08:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:14 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:08:14.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:14.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:14.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:14.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:14.794 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:14.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:14.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:14.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:14.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:15.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:15.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:15.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:15.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:15.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:15.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:15.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:15.463 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:15.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:15.591 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:15 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:08:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:15 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:08:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:15 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:08:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:15 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:08:15.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:08:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:15 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:08:15.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:15.861 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:15.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:15.963 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:16.000 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:16.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:16.131 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:16.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:16.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:16.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:16.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:16.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:16.523 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: osdmap e154: 8 total, 8 up, 8 in 2024-03-04T16:08:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: pgmap v224: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:16.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:08:16.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:16 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:08:16.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:16.665 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: osdmap e154: 8 total, 8 up, 8 in 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: pgmap v224: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: osdmap e154: 8 total, 8 up, 8 in 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: pgmap v224: 148 pgs: 148 active+clean; 982 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:08:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:16 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:08:16.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:16.810 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:16.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:16.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:16.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:17.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:17.073 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:17.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:17.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:17.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:17.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:17.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:17.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:08:17.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:08:17.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:08:17.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:08:17.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:08:17.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:08:17.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:08:17.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:17 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:08:17.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:17.481 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:17.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:17 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:08:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:17 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:08:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:17 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:08:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:17 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:08:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:17.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:17.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:17.762 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:17.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:17.892 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:17.921 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:17.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:18.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:18.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:18.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:18.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:18.216 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:18.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:18.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:18.350 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:08:18.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:18.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:18.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:08:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:08:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: pgmap v225: 148 pgs: 148 active+clean; 944 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-04T16:08:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:08:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:08:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:08:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:08:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3487669347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:18 smithi176 ceph-mon[33619]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: pgmap v225: 148 pgs: 148 active+clean; 944 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3487669347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[29779]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: pgmap v225: 148 pgs: 148 active+clean; 944 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 688 op/s 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3487669347' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.674 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:18 smithi038 ceph-mon[38113]: from='client.24943 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:18.703 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:08:18.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:18.749 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:08:18.759 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:18.798 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:08:18.845 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:08:18.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:18.885 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:08:18.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:18.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:18.924 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:08:18.963 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:08:18.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:19.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:08:19.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:08:19.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:08:19.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:08:19.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:08:19.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:08:19.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:08:19.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:08:19.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:08:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.039 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:19.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:19.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:19.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.187 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:19.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:19.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.328 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:19.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:19.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.458 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:19.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:19.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.606 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:19.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: osdmap e155: 8 total, 8 up, 8 in 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1605773262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:19 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:19.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: osdmap e155: 8 total, 8 up, 8 in 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1605773262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:19.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: from='client.24943 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: osdmap e155: 8 total, 8 up, 8 in 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1605773262' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:19.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:19 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:19.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:19.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:19.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:19.887 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:19.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:19.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:20.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:20.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:20.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:20.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:20.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:20.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:20.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:20.441 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: osdmap e156: 8 total, 8 up, 8 in 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: pgmap v228: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:08:20.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: osdmap e156: 8 total, 8 up, 8 in 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: pgmap v228: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:08:20.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:20 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:08:20.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:20.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.574 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:20.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:20.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:20.703 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:20.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:20.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:20.844 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: osdmap e156: 8 total, 8 up, 8 in 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: pgmap v228: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:08:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:20 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:08:20.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:20.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:20.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:21.077 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.107 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:21.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:21.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.235 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:21.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:21.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:21.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:21.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.500 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:21.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:21.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:21.648 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:21.733 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:08:21.733 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: osdmap e157: 8 total, 8 up, 8 in 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: 5.a deep-scrub starts 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[38113]: 5.a deep-scrub ok 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:21.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: osdmap e157: 8 total, 8 up, 8 in 2024-03-04T16:08:21.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: 5.a deep-scrub starts 2024-03-04T16:08:21.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:21 smithi038 ceph-mon[29779]: 5.a deep-scrub ok 2024-03-04T16:08:21.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:21.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:08:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:08:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: osdmap e157: 8 total, 8 up, 8 in 2024-03-04T16:08:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: 5.a deep-scrub starts 2024-03-04T16:08:21.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:21 smithi176 ceph-mon[33619]: 5.a deep-scrub ok 2024-03-04T16:08:21.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:21.970 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:21.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:22.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:22.098 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:22.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:22.228 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:22.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:22.330 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:22.355 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:22.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:22.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:22.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:08:22.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:08:22.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:22.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:22.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: osdmap e158: 8 total, 8 up, 8 in 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: pgmap v231: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 7.e scrub starts 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:08:22.576 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2586965080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[38113]: from='client.34946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: osdmap e158: 8 total, 8 up, 8 in 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: pgmap v231: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-04T16:08:22.577 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 7.e scrub starts 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2586965080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:22 smithi038 ceph-mon[29779]: from='client.34946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:08:22.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:22.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:22.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:22.780 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-04T16:08:22.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:08:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:08:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: osdmap e158: 8 total, 8 up, 8 in 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: pgmap v231: 148 pgs: 148 active+clean; 678 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 7.e scrub starts 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2586965080' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:22 smithi176 ceph-mon[33619]: from='client.34946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:08:22.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:22.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:22.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:22.933 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:08:22.933 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:08:23.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:23.068 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:23.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:23.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:23.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:23.484 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:23.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:23.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:23.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 7.e scrub ok 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:08:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 4.1a deep-scrub starts 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: 4.1a deep-scrub ok 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[29779]: osdmap e159: 8 total, 8 up, 8 in 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 7.e scrub ok 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:08:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 4.1a deep-scrub starts 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: 4.1a deep-scrub ok 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:08:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:23 smithi038 ceph-mon[38113]: osdmap e159: 8 total, 8 up, 8 in 2024-03-04T16:08:23.762 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:23.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:08:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:08:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:08:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 7.e scrub ok 2024-03-04T16:08:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 4.1a deep-scrub starts 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: 4.1a deep-scrub ok 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: from='client.34946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:08:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:23 smithi176 ceph-mon[33619]: osdmap e159: 8 total, 8 up, 8 in 2024-03-04T16:08:23.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:23.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:23.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:24.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:24.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:24.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:24.215 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:24.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:24.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:08:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.375 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:24.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 5 1 2024-03-04T16:08:24.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:24.499 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:24.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:24.574 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:24.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:24.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:08:24.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:08:24.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:08:24.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: pgmap v233: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.19 deep-scrub starts 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[29779]: 5.19 deep-scrub ok 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: pgmap v233: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.19 deep-scrub starts 2024-03-04T16:08:24.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:24 smithi038 ceph-mon[38113]: 5.19 deep-scrub ok 2024-03-04T16:08:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.662 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:24.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:24.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:24.791 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:08:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:08:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:08:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:08:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: pgmap v233: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-04T16:08:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.19 deep-scrub starts 2024-03-04T16:08:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:24 smithi176 ceph-mon[33619]: 5.19 deep-scrub ok 2024-03-04T16:08:24.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:24.926 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:24.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:25.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:25.054 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:25.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:25.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:25.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:25.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:25.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:25.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:25.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:25.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 4.3 deep-scrub starts 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 4.3 deep-scrub ok 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:08:25.686 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:08:25.687 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 4.3 deep-scrub starts 2024-03-04T16:08:25.687 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 4.3 deep-scrub ok 2024-03-04T16:08:25.687 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:08:25.687 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:25 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:08:25.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:25.761 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:08:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:08:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 4.3 deep-scrub starts 2024-03-04T16:08:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 4.3 deep-scrub ok 2024-03-04T16:08:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:08:25.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:25 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:08:25.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:25.905 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:25.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:26.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:26.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:26.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:26.186 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:26.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:26.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:26.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:26.445 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:26.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:26.583 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:26.687 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:26.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[29779]: 7.7 scrub starts 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[29779]: 7.7 scrub ok 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[29779]: pgmap v234: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 830 op/s 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[38113]: 7.7 scrub starts 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[38113]: 7.7 scrub ok 2024-03-04T16:08:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:26 smithi038 ceph-mon[38113]: pgmap v234: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 830 op/s 2024-03-04T16:08:26.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:26.839 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:26 smithi176 ceph-mon[33619]: 7.7 scrub starts 2024-03-04T16:08:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:26 smithi176 ceph-mon[33619]: 7.7 scrub ok 2024-03-04T16:08:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:26 smithi176 ceph-mon[33619]: pgmap v234: 148 pgs: 148 active+clean; 367 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 830 op/s 2024-03-04T16:08:26.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:26.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:26.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:27.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:27.106 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:27.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:27.248 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:27.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:27.382 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:27.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:27.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.549 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:27.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:27.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.682 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:27.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 7.1 scrub starts 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:08:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[29779]: 7.1 scrub ok 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 7.1 scrub starts 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:08:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:27 smithi038 ceph-mon[38113]: 7.1 scrub ok 2024-03-04T16:08:27.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:27.825 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 7.1 scrub starts 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:08:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:27 smithi176 ceph-mon[33619]: 7.1 scrub ok 2024-03-04T16:08:27.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:27.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:27.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:28.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:28.199 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:28.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:28.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:28.368 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:28.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:28.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:28.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:28.601 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Removed 118265 objects 2024-03-04T16:08:28.601 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Clean up completed and total clean up time :130.617 2024-03-04T16:08:28.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:28.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: pgmap v235: 148 pgs: 148 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 796 op/s 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: osdmap e160: 8 total, 8 up, 8 in 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:08:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: pgmap v235: 148 pgs: 148 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 796 op/s 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: osdmap e160: 8 total, 8 up, 8 in 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:08:28.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:08:28.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:28 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:08:28.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:28.757 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: pgmap v235: 148 pgs: 148 active+clean; 329 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 728 B/s rd, 0 B/s wr, 796 op/s 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: osdmap e160: 8 total, 8 up, 8 in 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:08:28.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:08:28.760 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:28 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:08:28.764 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:28.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:28.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:28.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:28.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:29.000 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:29.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:29.127 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:29.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:29.253 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:29.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:29.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:29.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:29.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:29.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:29.631 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:29.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 7.f scrub starts 2024-03-04T16:08:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[29779]: 7.f scrub ok 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 7.f scrub starts 2024-03-04T16:08:29.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:29 smithi038 ceph-mon[38113]: 7.f scrub ok 2024-03-04T16:08:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:29.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:08:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:08:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:08:29.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:08:29.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:08:29.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:08:29.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 7.f scrub starts 2024-03-04T16:08:29.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:29 smithi176 ceph-mon[33619]: 7.f scrub ok 2024-03-04T16:08:29.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:29.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:29.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:29.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:30.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:30.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:30.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:30.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:30.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:30.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:30.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:30.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:30.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:30.519 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:30.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:30.649 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:30.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:30 smithi038 ceph-mon[29779]: pgmap v237: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-04T16:08:30.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:30 smithi038 ceph-mon[38113]: pgmap v237: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-04T16:08:30.750 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:30.777 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:30.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:30 smithi176 ceph-mon[33619]: pgmap v237: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-04T16:08:30.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:30.903 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:30.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:31.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:31.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:31.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:31.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:31.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:31.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:31.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.417 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:31.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:31.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.545 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:31.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:31.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 3.1e deep-scrub starts 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 3.1e deep-scrub ok 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:08:31.656 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 3.1e deep-scrub starts 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 3.1e deep-scrub ok 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:08:31.657 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:31 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:08:31.673 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:31.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:31.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:31.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:08:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:08:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:08:31.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:08:31.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 3.1e deep-scrub starts 2024-03-04T16:08:31.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 3.1e deep-scrub ok 2024-03-04T16:08:31.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:08:31.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:31 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:08:31.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:31.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:31.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:32.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:32.058 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:32.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.184 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:32.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:32.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:32.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:32.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.439 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:32.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:32.492 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:08:32.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: pgmap v238: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 595 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:08:32.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:08:32.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:08:32.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/820879741' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[29779]: from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: pgmap v238: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 595 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/820879741' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:32 smithi038 ceph-mon[38113]: from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.494 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-04T16:08:32.540 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:32.565 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:32.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.692 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:32.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:32.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:32.824 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:32.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:32.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: pgmap v238: 148 pgs: 148 active+clean; 53 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 595 B/s rd, 0 B/s wr, 838 op/s 2024-03-04T16:08:32.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:08:32.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:08:32.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:08:32.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/820879741' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:32 smithi176 ceph-mon[33619]: from='client.34958 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:08:32.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:32.948 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:08:32.948 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:08:32.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:32.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:33.056 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.085 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:33.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:33.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:33.213 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:33.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:33.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:33.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:33.453 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:33.467 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:33.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:33.593 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:33.624 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-04T16:08:33.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-04T16:08:33.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.719 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:33.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[29779]: from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[29779]: osdmap e161: 8 total, 8 up, 8 in 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:08:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[38113]: from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:08:33.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[38113]: osdmap e161: 8 total, 8 up, 8 in 2024-03-04T16:08:33.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:33.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:33.849 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:33.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:33 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2933504287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:08:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:33 smithi176 ceph-mon[33619]: from='client.34958 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:08:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:33 smithi176 ceph-mon[33619]: osdmap e161: 8 total, 8 up, 8 in 2024-03-04T16:08:33.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:33.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 6 1 2024-03-04T16:08:33.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:33.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:33.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:34.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:34.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:34.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.231 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:34.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:34.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:34.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.360 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:34.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:34.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:34.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:34.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.614 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:34.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:34.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:08:34.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:08:34.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[29779]: pgmap v240: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s 2024-03-04T16:08:34.683 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:08:34.683 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:08:34.683 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:34 smithi038 ceph-mon[38113]: pgmap v240: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s 2024-03-04T16:08:34.716 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:34.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:34.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:34 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:08:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:34 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:08:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:34 smithi176 ceph-mon[33619]: pgmap v240: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 733 op/s 2024-03-04T16:08:34.870 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:08:34.873 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:34.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:34.893 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:08:34.915 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:08:34.935 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:08:34.954 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:08:34.974 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:08:34.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:34.994 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:08:35.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:35.014 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:08:35.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:08:35.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:35.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:35.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:35.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:35.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:08:35.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:35.382 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:35.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:35.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:35.607 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:35.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:35.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:35.645 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:35.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2184101601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:35.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2184101601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:35.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:35.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2184101601' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:35.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:35.891 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:35.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:35.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:36.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:36.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:36.145 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:36.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:36.274 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:36.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.405 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:36.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:36.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:08:36.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:36.533 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:36.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:36.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:36.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:36.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:36.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:37.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:37.043 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:37.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:37.172 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[29779]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s 2024-03-04T16:08:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[38113]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s 2024-03-04T16:08:37.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:37.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:37.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:37.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:37.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.431 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:37.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:37.500 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:37.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:37.559 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:37 smithi176 ceph-mon[33619]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 657 op/s 2024-03-04T16:08:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:37 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:37.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:37.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:37.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:37.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.813 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:37.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:37.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:37.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:37.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:38.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:38.065 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:38.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.196 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:38.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:38.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:38.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:38.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:38.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:38.451 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:38.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:38.577 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:38.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:38.707 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:38.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:08:38.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:38.836 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:38.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:38.965 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:38.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:39.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:39.094 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:39.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:39.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:39.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:39 smithi038 ceph-mon[29779]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 B/s wr, 406 op/s 2024-03-04T16:08:39.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:39 smithi038 ceph-mon[38113]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 B/s wr, 406 op/s 2024-03-04T16:08:39.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:39.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:39.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:39.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:39.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.604 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:39.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:39 smithi176 ceph-mon[33619]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 514 B/s rd, 0 B/s wr, 406 op/s 2024-03-04T16:08:39.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:39.728 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:39.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:39.854 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:39.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:39.979 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:39.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:40.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:40.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:40.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:40.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:40.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:40.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:40 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.604 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:40 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:40 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:40 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:40.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:40.730 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:40.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.856 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:40.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:40.957 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:40.982 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:40.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:41.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:41.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:41.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:41.232 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[38113]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:08:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[29779]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:08:41.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:41 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:41.362 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:41.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:41.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:41.492 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:41.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:41 smithi176 ceph-mon[33619]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:41 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:41 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:08:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:41 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:08:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:41 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.642 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:41.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:41.743 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:41.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:41.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:41.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:41.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:42.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:42.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:42.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:42.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:42.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:42.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:42.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:42.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[29779]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[29779]: osdmap e162: 8 total, 8 up, 8 in 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[38113]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:42.418 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:42 smithi038 ceph-mon[38113]: osdmap e162: 8 total, 8 up, 8 in 2024-03-04T16:08:42.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.533 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:42.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:42.594 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-mon[33619]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 85 op/s 2024-03-04T16:08:42.594 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:08:42.594 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:08:42.594 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:08:42.595 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-mon[33619]: osdmap e162: 8 total, 8 up, 8 in 2024-03-04T16:08:42.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.659 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:42.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:42.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:42.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:42.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:42.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:42.914 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:42.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:43.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:43.043 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:43.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:43.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:43.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.175 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:43.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1733107008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[29779]: from='client.34970 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1733107008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-mon[38113]: from='client.34970 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.248 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:08:43.276 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:43.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:43.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:43.425 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:43.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:43.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:43.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:43.574 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:08:43.574 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:08:43.574 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:08:43.575 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:08:43.575 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:08:43.575 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:43 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1733107008' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:43 smithi176 ceph-mon[33619]: from='client.34970 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:08:43.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:43.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:43.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:43.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:43.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:43.942 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:43.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:44.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:44.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:44.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:44.196 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:44.240 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-03-04T16:08:44.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:08:44.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:44.329 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:44.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:44.459 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:44.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[29779]: from='client.34970 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[29779]: osdmap e163: 8 total, 8 up, 8 in 2024-03-04T16:08:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[29779]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:08:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[38113]: from='client.34970 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[38113]: osdmap e163: 8 total, 8 up, 8 in 2024-03-04T16:08:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[38113]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:08:44.561 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:44.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:44 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:44 smithi176 ceph-mon[33619]: from='client.34970 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:08:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:44 smithi176 ceph-mon[33619]: osdmap e163: 8 total, 8 up, 8 in 2024-03-04T16:08:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:44 smithi176 ceph-mon[33619]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:08:44.687 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:44.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:44.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:44.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:44.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:44.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 7 1 2024-03-04T16:08:44.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:44.976 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:44.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:44.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:45.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.107 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:45.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:45.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:45.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:45.361 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:45.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:45.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:45.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:08:45.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:45 smithi038 ceph-mon[29779]: osdmap e164: 8 total, 8 up, 8 in 2024-03-04T16:08:45.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:45 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:08:45.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:45 smithi038 ceph-mon[38113]: osdmap e164: 8 total, 8 up, 8 in 2024-03-04T16:08:45.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:45 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3951341900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:08:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:45 smithi176 ceph-mon[33619]: osdmap e164: 8 total, 8 up, 8 in 2024-03-04T16:08:45.615 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:45.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.742 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:45.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:45.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:45.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:45.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:45.994 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:45.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:46.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.121 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:46.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:46.248 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:46.254 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_0' removed 2024-03-04T16:08:46.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:46.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:46.478 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:46 smithi038 ceph-mon[29779]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:08:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:46 smithi038 ceph-mon[38113]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:08:46.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:46.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:46.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:46 smithi176 ceph-mon[33619]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:08:46.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:08:46.631 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:46.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:46.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:46.859 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:46.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:46.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:46.961 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-04T16:08:46.962 INFO:tasks.radosbench:Beginning radosbench... 2024-03-04T16:08:46.962 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-04T16:08:46.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-04T16:08:46.968 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:46.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:47.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:47.114 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:47.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:47.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.276 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:47.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:47.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.401 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:47.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:47.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:08:47.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:47 smithi038 ceph-mon[29779]: osdmap e165: 8 total, 8 up, 8 in 2024-03-04T16:08:47.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:08:47.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:47 smithi038 ceph-mon[38113]: osdmap e165: 8 total, 8 up, 8 in 2024-03-04T16:08:47.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:47.529 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:47.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:08:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1146901454' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:08:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:47 smithi176 ceph-mon[33619]: osdmap e165: 8 total, 8 up, 8 in 2024-03-04T16:08:47.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:47.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:47.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:47.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:47.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:47.914 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:47.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:48.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.040 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:48.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:48.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:48.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:48.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:48.289 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:48.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:48.419 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:48.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:48.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:48 smithi038 ceph-mon[29779]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-03-04T16:08:48.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:48 smithi038 ceph-mon[38113]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-03-04T16:08:48.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:48 smithi176 ceph-mon[33619]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 727 B/s rd, 0 op/s 2024-03-04T16:08:48.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:48.670 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:08:48.676 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:48.689 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:08:48.708 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:08:48.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:08:48.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:08:48.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:08:48.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:48.789 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:08:48.805 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:48.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:48.808 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:08:48.810 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:08:48.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:48.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:48.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:49.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:49.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.186 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:49.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:49.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:49.316 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:49.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:49.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:49.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4040375397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:49.496 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4040375397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:49.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:49.573 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:49.595 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4040375397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:08:49.629 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:08:49.629 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:08:49.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.703 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:49.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:49.804 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.829 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:49.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:49.833 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:49.833 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:49.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:49.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:49.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:50.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.088 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:50.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:50.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:50.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:50.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:50.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:50.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:50.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:08:50.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:50.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:50.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:50.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:50.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.722 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:50.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:50.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:50.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:50.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:50.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:50.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:51.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:51.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:51.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:51 smithi038 ceph-mon[29779]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-04T16:08:51.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:51 smithi038 ceph-mon[38113]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-04T16:08:51.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:51.357 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:51.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:08:51.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:51.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:51.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:51 smithi176 ceph-mon[33619]: pgmap v252: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-04T16:08:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.615 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:51.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:51.739 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:51.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:51.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:51.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:51.998 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:51.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:52.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:52.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:52.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:52.249 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:52.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:52.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:52.478 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:52.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:52.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:52.505 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:52.605 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:08:52.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:52.631 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:52.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:52.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:52.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:08:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:08:52.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:52.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:52.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:52.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:53.115 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.144 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:53.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.269 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:53.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:53.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:53.398 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:53 smithi038 ceph-mon[29779]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:53.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:08:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:08:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:08:53.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:53 smithi038 ceph-mon[38113]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:53.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:53.527 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:53 smithi176 ceph-mon[33619]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:08:53.629 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.657 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:53.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:53.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:53.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:53.914 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:53.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:54.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:54.039 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:54.062 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:54.063 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:54.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:54.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:54.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:54.194 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:08:54.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:54.296 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:54.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.422 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3530592516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[29779]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3530592516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:54 smithi038 ceph-mon[38113]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:54.547 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:08:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3530592516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:54 smithi176 ceph-mon[33619]: from='client.34994 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:08:54.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:54.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.673 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:54.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:54.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:08:54.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:54.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:54.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:08:55.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:55.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:55.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:55.178 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:55.208 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_1' created 2024-03-04T16:08:55.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:55.306 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:55.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.436 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: osdmap e166: 8 total, 8 up, 8 in 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/156378424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[29779]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:55.437 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:55.438 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: osdmap e166: 8 total, 8 up, 8 in 2024-03-04T16:08:55.438 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/156378424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.438 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:55 smithi038 ceph-mon[38113]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:08:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:55.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:08:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-04T16:08:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:08:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: from='client.34994 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:08:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: osdmap e166: 8 total, 8 up, 8 in 2024-03-04T16:08:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/156378424' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:55 smithi176 ceph-mon[33619]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-04T16:08:55.662 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.692 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:55.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.822 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:55.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:55.950 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:55.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:56.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:56.080 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:56.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:56.209 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:56.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:56.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:56.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:56.468 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[29779]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[29779]: osdmap e167: 8 total, 8 up, 8 in 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[29779]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[38113]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-04T16:08:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[38113]: osdmap e167: 8 total, 8 up, 8 in 2024-03-04T16:08:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:56 smithi038 ceph-mon[38113]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:56.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:56.598 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:56 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:08:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:56 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:08:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:56 smithi176 ceph-mon[33619]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-04T16:08:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:56 smithi176 ceph-mon[33619]: osdmap e167: 8 total, 8 up, 8 in 2024-03-04T16:08:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:56 smithi176 ceph-mon[33619]: pgmap v257: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:08:56.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:56.725 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:56.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:56.851 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:56.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:56.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:56.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:57.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:57.109 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:57.211 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:57.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.238 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:57.244 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-04T16:08:57.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:57.366 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:57.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:08:57.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:08:57.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: osdmap e168: 8 total, 8 up, 8 in 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1954763861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[29779]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: osdmap e168: 8 total, 8 up, 8 in 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1954763861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:57 smithi038 ceph-mon[38113]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:57.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:08:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:08:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: osdmap e168: 8 total, 8 up, 8 in 2024-03-04T16:08:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1954763861' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:57 smithi176 ceph-mon[33619]: from='client.35006 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:08:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:57.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:57.753 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:57.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:08:57.879 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:57.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:08:57.981 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:58.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:58.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:58.131 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:58.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:58.258 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:58.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:58.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:58.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:58.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-04T16:08:58.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:58.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:58 smithi176 ceph-mon[33619]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:08:58.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:58 smithi176 ceph-mon[33619]: osdmap e169: 8 total, 8 up, 8 in 2024-03-04T16:08:58.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:58 smithi176 ceph-mon[33619]: pgmap v260: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-04T16:08:58.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:58 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:08:58.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:58 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:08:58.513 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:58.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:58.639 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:58.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:58.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[29779]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:08:58.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[29779]: osdmap e169: 8 total, 8 up, 8 in 2024-03-04T16:08:58.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[29779]: pgmap v260: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-04T16:08:58.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:08:58.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:08:58.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[38113]: from='client.35006 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:08:58.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[38113]: osdmap e169: 8 total, 8 up, 8 in 2024-03-04T16:08:58.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[38113]: pgmap v260: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-04T16:08:58.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:08:58.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:58 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:08:58.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:58.751 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:08:58.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:58.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:58.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:58.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:58.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:08:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.024 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:59.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:08:59.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:59.149 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:59.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:59.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:08:59.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:08:59.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:08:59.273 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:08:59.274 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:08:59.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:08:59.275 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:08:59 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:08:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.276 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:59.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:59.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:08:59.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:08:59.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:08:59.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:08:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:08:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:08:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:08:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:08:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:08:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:08:59 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:08:59.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:08:59.654 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:08:59.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:08:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:08:59.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:08:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:08:59.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:08:59.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:00.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:00.031 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:00.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:00.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:00.200 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:09:00.200 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:09:00.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:00.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[29779]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-04T16:09:00.360 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:09:00.361 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:00 smithi038 ceph-mon[38113]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-04T16:09:00.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:00.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:00.415 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:09:00.434 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:09:00.453 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:09:00.472 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:09:00.492 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:09:00.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:00.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:09:00.531 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:09:00.534 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:00.551 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:09:00.552 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:09:00.553 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:09:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:09:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:09:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:09:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:09:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:09:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:09:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:00 smithi176 ceph-mon[33619]: pgmap v261: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-04T16:09:00.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:00.659 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:00.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:00.784 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:00.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:00.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:00.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:01.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:01.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:01.160 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:01.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:01.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:01.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:01.411 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:01.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:01.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:01.539 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:01 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:09:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:01 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:09:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2304994119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:01.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.665 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:01.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:01.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:09:01.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:09:01.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2304994119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:01.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:09:01.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:09:01.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2304994119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:01.729 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T16:09:01.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:09:01.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:01.794 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:01.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:01.924 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:01.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:02.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:02.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.049 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:02.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:02.119 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:02.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:02.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:09:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:02.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:02.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:02.305 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:02.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:02.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:09:02.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:09:02.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:02 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:09:02.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:02.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:09:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:09:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:09:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:09:02.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:02.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:02.814 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:02.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:02.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:02.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:02.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:03.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:03.065 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:03.109 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:03.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:03.191 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:03.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:03.320 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:03.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:03.446 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:03.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:03.572 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:03 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:09:03.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:03 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:09:03.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:03.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:09:03.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:03.686 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:03.702 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:03.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.833 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:03.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:03.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:03.961 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:03.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:03.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:04.006 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:04.022 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:04.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:04.087 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:04.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:04.217 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:04.293 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:09:04.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.343 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:04.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:04.444 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:04.471 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[29779]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[38113]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:04.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:04 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:04.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:04.597 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:04 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:09:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:04 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:09:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:04 smithi176 ceph-mon[33619]: pgmap v263: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:04 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:04.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.727 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:04.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:04.809 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:09:04.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.857 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:04.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:04.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:04.987 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:04.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:05.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:05.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:05.214 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:05.240 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:05.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:05.366 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:05.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:05.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:05.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:09:05.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[29779]: osdmap e170: 8 total, 8 up, 8 in 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:09:05.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:05.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:05 smithi038 ceph-mon[38113]: osdmap e170: 8 total, 8 up, 8 in 2024-03-04T16:09:05.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:09:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:09:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:09:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:09:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:09:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:09:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:09:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1944808465' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:05 smithi176 ceph-mon[33619]: osdmap e170: 8 total, 8 up, 8 in 2024-03-04T16:09:05.616 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:05.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:05.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:05.867 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:05.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:05.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:05.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:06.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:06.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:06.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:06.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:06.302 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-04T16:09:06.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:06.376 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:06.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:06.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.503 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:06.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:06.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[29779]: 4.1b deep-scrub starts 2024-03-04T16:09:06.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[29779]: 4.1b deep-scrub ok 2024-03-04T16:09:06.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[29779]: pgmap v265: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:06.599 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:09:06.599 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[38113]: 4.1b deep-scrub starts 2024-03-04T16:09:06.600 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[38113]: 4.1b deep-scrub ok 2024-03-04T16:09:06.600 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[38113]: pgmap v265: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:06.600 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:09:06.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:06 smithi176 ceph-mon[33619]: 4.1b deep-scrub starts 2024-03-04T16:09:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:06 smithi176 ceph-mon[33619]: 4.1b deep-scrub ok 2024-03-04T16:09:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:06 smithi176 ceph-mon[33619]: pgmap v265: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:09:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:09:06.628 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:06.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:06.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:06.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:06.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:06.879 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:06.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:06.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:06.981 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:07.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:07.073 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:07.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:07.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.261 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:07.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:07.386 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:07.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:07.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: osdmap e171: 8 total, 8 up, 8 in 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:09:07.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:09:07.580 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: osdmap e171: 8 total, 8 up, 8 in 2024-03-04T16:09:07.580 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:07.580 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:09:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:09:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:09:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:09:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3860668210' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:09:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: osdmap e171: 8 total, 8 up, 8 in 2024-03-04T16:09:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:07.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:09:07.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:07.639 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:07.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:07.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:07.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:07.899 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:07.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:08.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:08.008 DEBUG:teuthology.orchestra.run.smithi038:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-04T16:09:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:08.026 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-04T16:09:08.026 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-04T16:09:08.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:hints = 1 2024-03-04T16:09:08.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-04T16:09:08.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object prefix: benchmark_data_smithi038_107057 2024-03-04T16:09:08.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:09:08.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 0 0 0 0 0 0 - 0 2024-03-04T16:09:08.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:08.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:08.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[29779]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:09:08.383 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:09:08.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:08 smithi038 ceph-mon[38113]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:09:08.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:08.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:08.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:09:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:08 smithi176 ceph-mon[33619]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:09:08.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:08.745 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:08.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:08.878 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:08.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:08.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:09.015 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:09.074 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 1 16 3401 3385 211.55 211.562 0.00265119 0.00471191 2024-03-04T16:09:09.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:09.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:09.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:09.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:09.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.436 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:09.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:09.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:09.579 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:09:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:09:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:09.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:09:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:09:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:09 smithi176 ceph-mon[33619]: osdmap e172: 8 total, 8 up, 8 in 2024-03-04T16:09:09.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:09.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:09:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[29779]: osdmap e172: 8 total, 8 up, 8 in 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:09:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:09 smithi038 ceph-mon[38113]: osdmap e172: 8 total, 8 up, 8 in 2024-03-04T16:09:09.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:09.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:09.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:09.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:10.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:10.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 2 16 6715 6699 209.244 207.125 0.00503252 0.00477111 2024-03-04T16:09:10.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.157 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:10.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:10.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:10.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:10.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:10.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[29779]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[29779]: 6.10 deep-scrub starts 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[29779]: 6.10 deep-scrub ok 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[38113]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:10.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[38113]: 6.10 deep-scrub starts 2024-03-04T16:09:10.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:10 smithi038 ceph-mon[38113]: 6.10 deep-scrub ok 2024-03-04T16:09:10.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:10.579 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:10 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:09:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:10 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:09:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:10 smithi176 ceph-mon[33619]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:09:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:10 smithi176 ceph-mon[33619]: 6.10 deep-scrub starts 2024-03-04T16:09:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:10 smithi176 ceph-mon[33619]: 6.10 deep-scrub ok 2024-03-04T16:09:10.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:10.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:10.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:10.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:10.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:10.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:10.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:11.014 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:11.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 3 16 9850 9834 204.804 195.938 0.00294627 0.00487459 2024-03-04T16:09:11.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.168 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:11.203 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:09:11.204 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:09:11.204 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:09:11.204 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:09:11.204 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:09:11.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:11.312 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:11.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:11.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:11 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:09:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:11 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:09:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:11 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:09:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:11 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:09:11.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.600 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:11 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:09:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:11 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:09:11.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:11.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:11.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:11.875 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:11.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:11.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:12.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:12.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:09:12.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 4 16 12589 12573 196.397 171.188 0.00455811 0.00508259 2024-03-04T16:09:12.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:12.142 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:12.244 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:12.276 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:12.315 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:12.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:12.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:12.436 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:09:12.476 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:09:12.507 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:09:12.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:12.534 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:09:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:09:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-mon[33619]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 718 B/s rd, 0 op/s 2024-03-04T16:09:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:09:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:09:12.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:12.643 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[38113]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 718 B/s rd, 0 op/s 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:09:12.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:09:12.645 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[29779]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 718 B/s rd, 0 op/s 2024-03-04T16:09:12.645 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:09:12.645 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:12 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:09:12.682 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:12.687 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:09:12.729 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:09:12.771 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:09:12.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:12.817 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:09:12.825 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:09:12.826 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:09:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:12.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:12.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:12.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:12.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:12.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:13.056 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:13.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 5 16 14707 14691 183.592 132.375 0.00259498 0.00543973 2024-03-04T16:09:13.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:13.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:13.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:13.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:13.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:13.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:13.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1018829417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:13.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1018829417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:13.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:13.504 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:13.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:09:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:09:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:09:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:09:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1018829417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:13.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:13.645 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:13.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:13.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.771 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:13.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:13.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:13.908 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:13.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:14.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:14.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:14.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 6 16 17207 17191 179.034 156.25 0.00535068 0.00557932 2024-03-04T16:09:14.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:14.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:14.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:14.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:14.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:14.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:14.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:14 smithi176 ceph-mon[33619]: pgmap v271: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:09:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:14 smithi176 ceph-mon[33619]: 4.12 deep-scrub starts 2024-03-04T16:09:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:14 smithi176 ceph-mon[33619]: 4.12 deep-scrub ok 2024-03-04T16:09:14.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:14.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[29779]: pgmap v271: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:09:14.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[29779]: 4.12 deep-scrub starts 2024-03-04T16:09:14.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[29779]: 4.12 deep-scrub ok 2024-03-04T16:09:14.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[38113]: pgmap v271: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:09:14.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[38113]: 4.12 deep-scrub starts 2024-03-04T16:09:14.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:14 smithi038 ceph-mon[38113]: 4.12 deep-scrub ok 2024-03-04T16:09:14.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:14.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:14.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:14.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:14.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:14.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:14.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:15.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:15.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:15.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 7 16 19450 19434 173.483 140.188 0.0112896 0.00575799 2024-03-04T16:09:15.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:15.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:15.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:15.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:15.387 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:15.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:15.426 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:09:15.436 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:09:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:15.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:15.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:15.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:15.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:15.722 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:09:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:15 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:09:15.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:15 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:09:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:15 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:09:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:15 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:09:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:15 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:09:15.862 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:15.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:15.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.011 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:16.075 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 8 16 21394 21378 166.984 121.5 0.00143957 0.00594496 2024-03-04T16:09:16.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:16.143 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:16.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:16.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:16.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:16.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:09:16.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.570 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:16.671 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:16.732 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: 5.16 deep-scrub starts 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: 5.16 deep-scrub ok 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: from='client.15183 v1:172.21.15.38:0/883874821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[29779]: pgmap v272: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:09:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:09:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:09:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: 5.16 deep-scrub starts 2024-03-04T16:09:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: 5.16 deep-scrub ok 2024-03-04T16:09:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: from='client.15183 v1:172.21.15.38:0/883874821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:16 smithi038 ceph-mon[38113]: pgmap v272: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:09:16.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: 5.16 deep-scrub starts 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: 5.16 deep-scrub ok 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: from='client.15183 v1:172.21.15.38:0/883874821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:16 smithi176 ceph-mon[33619]: pgmap v272: 148 pgs: 148 active+clean; 867 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:09:16.875 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:16.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:16.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:16.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.018 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:17.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:17.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 9 15 23423 23408 162.522 126.875 0.0134182 0.00614616 2024-03-04T16:09:17.119 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.153 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:17.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:17.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:17.295 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:17.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:17.431 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:17 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:09:17.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:17 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:09:17.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:17 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:09:17.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:17 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:09:17.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.574 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:17.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:17.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.721 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:17.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:17.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:17 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:09:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:17 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:09:17.862 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:17.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:17.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.004 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:18.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:18.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 10 16 25883 25867 161.637 153.688 0.00994328 0.00618111 2024-03-04T16:09:18.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:18.146 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:18.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:18.305 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:18.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.443 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:18.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[38113]: pgmap v273: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:09:18.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:18 smithi038 ceph-mon[29779]: pgmap v273: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:09:18.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.580 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:18.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:18.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.711 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:18.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:18.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:18.853 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:18.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:18 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:09:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:18 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:09:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:18 smithi176 ceph-mon[33619]: pgmap v273: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:09:18.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:18.979 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:18.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:19.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 11 16 27868 27852 158.22 124.062 0.00454043 0.00631437 2024-03-04T16:09:19.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:19.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:19.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:19.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:19.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:19.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:19.481 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:19.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:19.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:19.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:19.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:09:19.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:09:19.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:09:19.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:09:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:09:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:09:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:09:19.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:19 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:09:19.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:19.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:19 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:09:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:19 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:09:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:19 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:09:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:19 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:09:19.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:19.921 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:19.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:20.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:20.047 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:20.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 12 16 29718 29702 154.67 115.625 0.00255944 0.00643315 2024-03-04T16:09:20.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:20.185 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:20.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:20.321 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:20.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:20.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:20.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.604 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:20.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:20.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.740 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:20.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:20.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:09:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:09:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[38113]: pgmap v274: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 922 B/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-04T16:09:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:09:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:09:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:20 smithi038 ceph-mon[29779]: pgmap v274: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 922 B/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-04T16:09:20.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:20 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:09:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:20 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:09:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:20 smithi176 ceph-mon[33619]: pgmap v274: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 922 B/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-04T16:09:20.878 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:20.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:20.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:20.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.013 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:21.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 13 16 31600 31584 151.82 117.625 0.00662003 0.00658204 2024-03-04T16:09:21.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:21.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:21.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:21.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:21.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.425 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:21.526 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:21.552 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:21.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[29779]: 6.12 deep-scrub starts 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[29779]: 6.12 deep-scrub ok 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[38113]: 6.12 deep-scrub starts 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[38113]: 6.12 deep-scrub ok 2024-03-04T16:09:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:21 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:09:21.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:21.840 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:21 smithi176 ceph-mon[33619]: 6.12 deep-scrub starts 2024-03-04T16:09:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:21 smithi176 ceph-mon[33619]: 6.12 deep-scrub ok 2024-03-04T16:09:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:21 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:09:21.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:21.981 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:21.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:21.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:22.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 14 15 33392 33377 148.98 112.062 0.0124029 0.00670652 2024-03-04T16:09:22.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:22.123 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-03-04T16:09:13.415815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":909449566,"num_objects":14066,"num_object_clones":0,"num_object_copies":42198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14066,"num_whiteouts":0,"num_read":1305,"num_read_kb":1151,"num_write":14483,"num_write_kb":890013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2956616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14525,"ondisk_log_size":14525,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3682236,"kb_used_data":2398524,"kb_used_omap":12,"kb_used_meta":1283571,"kb_avail":746278980,"statfs":{"total":767960285184,"available":764189675520,"internally_reserved":0,"allocated":2456088576,"data_stored":2452734298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1314377266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,6,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":908853248,"num_objects":13868,"num_object_clones":0,"num_object_copies":41604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13868,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13868,"num_write_kb":887552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002222"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320824+0000","last_change":"2024-03-04T16:06:46.979579+0000","last_active":"2024-03-04T16:09:08.320824+0000","last_peered":"2024-03-04T16:09:08.320824+0000","last_clean":"2024-03-04T16:09:08.320824+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:09:08.320824+0000","last_undegraded":"2024-03-04T16:09:08.320824+0000","last_fullsized":"2024-03-04T16:09:08.320824+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:46.979459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:46.979459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328883+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:09:08.328883+0000","last_peered":"2024-03-04T16:09:08.328883+0000","last_clean":"2024-03-04T16:09:08.328883+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:09:08.328883+0000","last_undegraded":"2024-03-04T16:09:08.328883+0000","last_fullsized":"2024-03-04T16:09:08.328883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:27.323986+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.329785+0000","last_change":"2024-03-04T16:08:56.062397+0000","last_active":"2024-03-04T16:09:08.329785+0000","last_peered":"2024-03-04T16:09:08.329785+0000","last_clean":"2024-03-04T16:09:08.329785+0000","last_became_active":"2024-03-04T16:07:36.205930+0000","last_became_peered":"2024-03-04T16:07:36.205930+0000","last_unstale":"2024-03-04T16:09:08.329785+0000","last_undegraded":"2024-03-04T16:09:08.329785+0000","last_fullsized":"2024-03-04T16:09:08.329785+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.471716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":257,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320540+0000","last_change":"2024-03-04T16:08:15.752830+0000","last_active":"2024-03-04T16:09:08.320540+0000","last_peered":"2024-03-04T16:09:08.320540+0000","last_clean":"2024-03-04T16:09:08.320540+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:09:08.320540+0000","last_undegraded":"2024-03-04T16:09:08.320540+0000","last_fullsized":"2024-03-04T16:09:08.320540+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:15.752737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.752737+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:22.985140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":225,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326002+0000","last_change":"2024-03-04T16:05:46.313428+0000","last_active":"2024-03-04T16:09:08.326002+0000","last_peered":"2024-03-04T16:09:08.326002+0000","last_clean":"2024-03-04T16:09:08.326002+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:09:08.326002+0000","last_undegraded":"2024-03-04T16:09:08.326002+0000","last_fullsized":"2024-03-04T16:09:08.326002+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:05:46.313300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320586+0000","last_change":"2024-03-04T16:09:00.760314+0000","last_active":"2024-03-04T16:09:08.320586+0000","last_peered":"2024-03-04T16:09:08.320586+0000","last_clean":"2024-03-04T16:09:08.320586+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:09:08.320586+0000","last_undegraded":"2024-03-04T16:09:08.320586+0000","last_fullsized":"2024-03-04T16:09:08.320586+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326262+0000","last_change":"2024-03-04T16:08:30.752343+0000","last_active":"2024-03-04T16:09:08.326262+0000","last_peered":"2024-03-04T16:09:08.326262+0000","last_clean":"2024-03-04T16:09:08.326262+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:09:08.326262+0000","last_undegraded":"2024-03-04T16:09:08.326262+0000","last_fullsized":"2024-03-04T16:09:08.326262+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.745969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":310,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325290+0000","last_change":"2024-03-04T16:08:21.634801+0000","last_active":"2024-03-04T16:09:08.325290+0000","last_peered":"2024-03-04T16:09:08.325290+0000","last_clean":"2024-03-04T16:09:08.325290+0000","last_became_active":"2024-03-04T16:07:36.221904+0000","last_became_peered":"2024-03-04T16:07:36.221904+0000","last_unstale":"2024-03-04T16:09:08.325290+0000","last_undegraded":"2024-03-04T16:09:08.325290+0000","last_fullsized":"2024-03-04T16:09:08.325290+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.436614+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":89,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.270128+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:09:12.270128+0000","last_peered":"2024-03-04T16:09:12.270128+0000","last_clean":"2024-03-04T16:09:12.270128+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:09:12.270128+0000","last_undegraded":"2024-03-04T16:09:12.270128+0000","last_fullsized":"2024-03-04T16:09:12.270128+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:35.559365+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326027+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:09:08.326027+0000","last_peered":"2024-03-04T16:09:08.326027+0000","last_clean":"2024-03-04T16:09:08.326027+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:09:08.326027+0000","last_undegraded":"2024-03-04T16:09:08.326027+0000","last_fullsized":"2024-03-04T16:09:08.326027+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.564603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327638+0000","last_change":"2024-03-04T16:08:18.150034+0000","last_active":"2024-03-04T16:09:08.327638+0000","last_peered":"2024-03-04T16:09:08.327638+0000","last_clean":"2024-03-04T16:09:08.327638+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:09:08.327638+0000","last_undegraded":"2024-03-04T16:09:08.327638+0000","last_fullsized":"2024-03-04T16:09:08.327638+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.921681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":293,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321069+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:09:08.321069+0000","last_peered":"2024-03-04T16:09:08.321069+0000","last_clean":"2024-03-04T16:09:08.321069+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:09:08.321069+0000","last_undegraded":"2024-03-04T16:09:08.321069+0000","last_fullsized":"2024-03-04T16:09:08.321069+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:24.124141+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.324859+0000","last_change":"2024-03-04T16:08:27.210748+0000","last_active":"2024-03-04T16:09:08.324859+0000","last_peered":"2024-03-04T16:09:08.324859+0000","last_clean":"2024-03-04T16:09:08.324859+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:09:08.324859+0000","last_undegraded":"2024-03-04T16:09:08.324859+0000","last_fullsized":"2024-03-04T16:09:08.324859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.210666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.210666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.062241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322752+0000","last_change":"2024-03-04T16:08:22.797116+0000","last_active":"2024-03-04T16:09:08.322752+0000","last_peered":"2024-03-04T16:09:08.322752+0000","last_clean":"2024-03-04T16:09:08.322752+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:09:08.322752+0000","last_undegraded":"2024-03-04T16:09:08.322752+0000","last_fullsized":"2024-03-04T16:09:08.322752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.796991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.796991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.136282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327353+0000","last_change":"2024-03-04T16:08:07.153510+0000","last_active":"2024-03-04T16:09:08.327353+0000","last_peered":"2024-03-04T16:09:08.327353+0000","last_clean":"2024-03-04T16:09:08.327353+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:09:08.327353+0000","last_undegraded":"2024-03-04T16:09:08.327353+0000","last_fullsized":"2024-03-04T16:09:08.327353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:07.153434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.153434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:27.242156+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":265,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321649+0000","last_change":"2024-03-04T16:09:04.791187+0000","last_active":"2024-03-04T16:09:08.321649+0000","last_peered":"2024-03-04T16:09:08.321649+0000","last_clean":"2024-03-04T16:09:08.321649+0000","last_became_active":"2024-03-04T16:04:22.732082+0000","last_became_peered":"2024-03-04T16:04:22.732082+0000","last_unstale":"2024-03-04T16:09:08.321649+0000","last_undegraded":"2024-03-04T16:09:08.321649+0000","last_fullsized":"2024-03-04T16:09:08.321649+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:14.516520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321170+0000","last_change":"2024-03-04T16:09:03.790808+0000","last_active":"2024-03-04T16:09:08.321170+0000","last_peered":"2024-03-04T16:09:08.321170+0000","last_clean":"2024-03-04T16:09:08.321170+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:09:08.321170+0000","last_undegraded":"2024-03-04T16:09:08.321170+0000","last_fullsized":"2024-03-04T16:09:08.321170+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:13.192635+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328732+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:09:08.328732+0000","last_peered":"2024-03-04T16:09:08.328732+0000","last_clean":"2024-03-04T16:09:08.328732+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:09:08.328732+0000","last_undegraded":"2024-03-04T16:09:08.328732+0000","last_fullsized":"2024-03-04T16:09:08.328732+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.853952+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.207691+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:09:11.207691+0000","last_peered":"2024-03-04T16:09:11.207691+0000","last_clean":"2024-03-04T16:09:11.207691+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:09:11.207691+0000","last_undegraded":"2024-03-04T16:09:11.207691+0000","last_fullsized":"2024-03-04T16:09:11.207691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.718715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":274,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321193+0000","last_change":"2024-03-04T16:08:27.760340+0000","last_active":"2024-03-04T16:09:08.321193+0000","last_peered":"2024-03-04T16:09:08.321193+0000","last_clean":"2024-03-04T16:09:08.321193+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:09:08.321193+0000","last_undegraded":"2024-03-04T16:09:08.321193+0000","last_fullsized":"2024-03-04T16:09:08.321193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:27.760194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.760194+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.308820+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":226,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326848+0000","last_change":"2024-03-04T16:06:26.278970+0000","last_active":"2024-03-04T16:09:08.326848+0000","last_peered":"2024-03-04T16:09:08.326848+0000","last_clean":"2024-03-04T16:09:08.326848+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:09:08.326848+0000","last_undegraded":"2024-03-04T16:09:08.326848+0000","last_fullsized":"2024-03-04T16:09:08.326848+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:26.278824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:26.278824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321847+0000","last_change":"2024-03-04T16:08:04.014351+0000","last_active":"2024-03-04T16:09:08.321847+0000","last_peered":"2024-03-04T16:09:08.321847+0000","last_clean":"2024-03-04T16:09:08.321847+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:09:08.321847+0000","last_undegraded":"2024-03-04T16:09:08.321847+0000","last_fullsized":"2024-03-04T16:09:08.321847+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:04.014236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.014236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:20.495511+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321739+0000","last_change":"2024-03-04T16:07:16.179561+0000","last_active":"2024-03-04T16:09:08.321739+0000","last_peered":"2024-03-04T16:09:08.321739+0000","last_clean":"2024-03-04T16:09:08.321739+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:09:08.321739+0000","last_undegraded":"2024-03-04T16:09:08.321739+0000","last_fullsized":"2024-03-04T16:09:08.321739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:16.179457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:07:16.179457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326241+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:09:08.326241+0000","last_peered":"2024-03-04T16:09:08.326241+0000","last_clean":"2024-03-04T16:09:08.326241+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:09:08.326241+0000","last_undegraded":"2024-03-04T16:09:08.326241+0000","last_fullsized":"2024-03-04T16:09:08.326241+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.113960+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322949+0000","last_change":"2024-03-04T16:09:01.793018+0000","last_active":"2024-03-04T16:09:08.322949+0000","last_peered":"2024-03-04T16:09:08.322949+0000","last_clean":"2024-03-04T16:09:08.322949+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:09:08.322949+0000","last_undegraded":"2024-03-04T16:09:08.322949+0000","last_fullsized":"2024-03-04T16:09:08.322949+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:09:01.792870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.792870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:29.041563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"124'9","reported_seq":336,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326315+0000","last_change":"2024-03-04T16:08:20.850258+0000","last_active":"2024-03-04T16:09:08.326315+0000","last_peered":"2024-03-04T16:09:08.326315+0000","last_clean":"2024-03-04T16:09:08.326315+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:09:08.326315+0000","last_undegraded":"2024-03-04T16:09:08.326315+0000","last_fullsized":"2024-03-04T16:09:08.326315+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:08:20.850137+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:06:42.058461+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.850137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.862934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328987+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:09:08.328987+0000","last_peered":"2024-03-04T16:09:08.328987+0000","last_clean":"2024-03-04T16:09:08.328987+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:09:08.328987+0000","last_undegraded":"2024-03-04T16:09:08.328987+0000","last_fullsized":"2024-03-04T16:09:08.328987+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:25.074925+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":93,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330082+0000","last_change":"2024-03-04T16:08:04.989047+0000","last_active":"2024-03-04T16:09:08.330082+0000","last_peered":"2024-03-04T16:09:08.330082+0000","last_clean":"2024-03-04T16:09:08.330082+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:09:08.330082+0000","last_undegraded":"2024-03-04T16:09:08.330082+0000","last_fullsized":"2024-03-04T16:09:08.330082+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:08:04.988877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.988877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:21.940131+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328877+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:09:08.328877+0000","last_peered":"2024-03-04T16:09:08.328877+0000","last_clean":"2024-03-04T16:09:08.328877+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:09:08.328877+0000","last_undegraded":"2024-03-04T16:09:08.328877+0000","last_fullsized":"2024-03-04T16:09:08.328877+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.336787+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330292+0000","last_change":"2024-03-04T16:09:01.127663+0000","last_active":"2024-03-04T16:09:08.330292+0000","last_peered":"2024-03-04T16:09:08.330292+0000","last_clean":"2024-03-04T16:09:08.330292+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:09:08.330292+0000","last_undegraded":"2024-03-04T16:09:08.330292+0000","last_fullsized":"2024-03-04T16:09:08.330292+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:01.883191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":257,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325524+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:09:08.325524+0000","last_peered":"2024-03-04T16:09:08.325524+0000","last_clean":"2024-03-04T16:09:08.325524+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:09:08.325524+0000","last_undegraded":"2024-03-04T16:09:08.325524+0000","last_fullsized":"2024-03-04T16:09:08.325524+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.074160+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":287,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326346+0000","last_change":"2024-03-04T16:08:58.094226+0000","last_active":"2024-03-04T16:09:08.326346+0000","last_peered":"2024-03-04T16:09:08.326346+0000","last_clean":"2024-03-04T16:09:08.326346+0000","last_became_active":"2024-03-04T16:04:22.734590+0000","last_became_peered":"2024-03-04T16:04:22.734590+0000","last_unstale":"2024-03-04T16:09:08.326346+0000","last_undegraded":"2024-03-04T16:09:08.326346+0000","last_fullsized":"2024-03-04T16:09:08.326346+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.639004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322655+0000","last_change":"2024-03-04T16:06:47.454119+0000","last_active":"2024-03-04T16:09:08.322655+0000","last_peered":"2024-03-04T16:09:08.322655+0000","last_clean":"2024-03-04T16:09:08.322655+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:09:08.322655+0000","last_undegraded":"2024-03-04T16:09:08.322655+0000","last_fullsized":"2024-03-04T16:09:08.322655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:47.454001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:47.454001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325318+0000","last_change":"2024-03-04T16:08:56.035409+0000","last_active":"2024-03-04T16:09:08.325318+0000","last_peered":"2024-03-04T16:09:08.325318+0000","last_clean":"2024-03-04T16:09:08.325318+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:09:08.325318+0000","last_undegraded":"2024-03-04T16:09:08.325318+0000","last_fullsized":"2024-03-04T16:09:08.325318+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.644040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.329692+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:09:08.329692+0000","last_peered":"2024-03-04T16:09:08.329692+0000","last_clean":"2024-03-04T16:09:08.329692+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:09:08.329692+0000","last_undegraded":"2024-03-04T16:09:08.329692+0000","last_fullsized":"2024-03-04T16:09:08.329692+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.356777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":239,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325627+0000","last_change":"2024-03-04T16:06:24.244107+0000","last_active":"2024-03-04T16:09:08.325627+0000","last_peered":"2024-03-04T16:09:08.325627+0000","last_clean":"2024-03-04T16:09:08.325627+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:09:08.325627+0000","last_undegraded":"2024-03-04T16:09:08.325627+0000","last_fullsized":"2024-03-04T16:09:08.325627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:06:24.243967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:24.243967+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326588+0000","last_change":"2024-03-04T16:08:06.354795+0000","last_active":"2024-03-04T16:09:08.326588+0000","last_peered":"2024-03-04T16:09:08.326588+0000","last_clean":"2024-03-04T16:09:08.326588+0000","last_became_active":"2024-03-04T16:07:36.224625+0000","last_became_peered":"2024-03-04T16:07:36.224625+0000","last_unstale":"2024-03-04T16:09:08.326588+0000","last_undegraded":"2024-03-04T16:09:08.326588+0000","last_fullsized":"2024-03-04T16:09:08.326588+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:23.563447+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322691+0000","last_change":"2024-03-04T16:08:58.788323+0000","last_active":"2024-03-04T16:09:08.322691+0000","last_peered":"2024-03-04T16:09:08.322691+0000","last_clean":"2024-03-04T16:09:08.322691+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:09:08.322691+0000","last_undegraded":"2024-03-04T16:09:08.322691+0000","last_fullsized":"2024-03-04T16:09:08.322691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.788235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.788235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326572+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:09:08.326572+0000","last_peered":"2024-03-04T16:09:08.326572+0000","last_clean":"2024-03-04T16:09:08.326572+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:09:08.326572+0000","last_undegraded":"2024-03-04T16:09:08.326572+0000","last_fullsized":"2024-03-04T16:09:08.326572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.344720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325185+0000","last_change":"2024-03-04T16:06:34.283125+0000","last_active":"2024-03-04T16:09:08.325185+0000","last_peered":"2024-03-04T16:09:08.325185+0000","last_clean":"2024-03-04T16:09:08.325185+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:09:08.325185+0000","last_undegraded":"2024-03-04T16:09:08.325185+0000","last_fullsized":"2024-03-04T16:09:08.325185+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:06:34.282977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:34.282977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.176010+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:09:10.176010+0000","last_peered":"2024-03-04T16:09:10.176010+0000","last_clean":"2024-03-04T16:09:10.176010+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:09:10.176010+0000","last_undegraded":"2024-03-04T16:09:10.176010+0000","last_fullsized":"2024-03-04T16:09:10.176010+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.303288+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325895+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:09:08.325895+0000","last_peered":"2024-03-04T16:09:08.325895+0000","last_clean":"2024-03-04T16:09:08.325895+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:09:08.325895+0000","last_undegraded":"2024-03-04T16:09:08.325895+0000","last_fullsized":"2024-03-04T16:09:08.325895+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:57.871591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323171+0000","last_change":"2024-03-04T16:06:36.455061+0000","last_active":"2024-03-04T16:09:08.323171+0000","last_peered":"2024-03-04T16:09:08.323171+0000","last_clean":"2024-03-04T16:09:08.323171+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:09:08.323171+0000","last_undegraded":"2024-03-04T16:09:08.323171+0000","last_fullsized":"2024-03-04T16:09:08.323171+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:36.454957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:06:36.454957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":261,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330210+0000","last_change":"2024-03-04T16:08:08.737650+0000","last_active":"2024-03-04T16:09:08.330210+0000","last_peered":"2024-03-04T16:09:08.330210+0000","last_clean":"2024-03-04T16:09:08.330210+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:09:08.330210+0000","last_undegraded":"2024-03-04T16:09:08.330210+0000","last_fullsized":"2024-03-04T16:09:08.330210+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:08.737521+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:04:07.040136+0000","last_clean_scrub_stamp":"2024-03-04T16:08:08.737521+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:13.356312+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.324321+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:09:08.324321+0000","last_peered":"2024-03-04T16:09:08.324321+0000","last_clean":"2024-03-04T16:09:08.324321+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:09:08.324321+0000","last_undegraded":"2024-03-04T16:09:08.324321+0000","last_fullsized":"2024-03-04T16:09:08.324321+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.513501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330111+0000","last_change":"2024-03-04T16:08:10.736897+0000","last_active":"2024-03-04T16:09:08.330111+0000","last_peered":"2024-03-04T16:09:08.330111+0000","last_clean":"2024-03-04T16:09:08.330111+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:09:08.330111+0000","last_undegraded":"2024-03-04T16:09:08.330111+0000","last_fullsized":"2024-03-04T16:09:08.330111+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.736771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.736771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:18.525211+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321480+0000","last_change":"2024-03-04T16:07:31.213593+0000","last_active":"2024-03-04T16:09:08.321480+0000","last_peered":"2024-03-04T16:09:08.321480+0000","last_clean":"2024-03-04T16:09:08.321480+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:09:08.321480+0000","last_undegraded":"2024-03-04T16:09:08.321480+0000","last_fullsized":"2024-03-04T16:09:08.321480+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:31.213444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:07:31.213444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321480+0000","last_change":"2024-03-04T16:09:02.785333+0000","last_active":"2024-03-04T16:09:08.321480+0000","last_peered":"2024-03-04T16:09:08.321480+0000","last_clean":"2024-03-04T16:09:08.321480+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:09:08.321480+0000","last_undegraded":"2024-03-04T16:09:08.321480+0000","last_fullsized":"2024-03-04T16:09:08.321480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.439534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":114,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:11.858799+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:09:11.858799+0000","last_peered":"2024-03-04T16:09:11.858799+0000","last_clean":"2024-03-04T16:09:11.858799+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:09:11.858799+0000","last_undegraded":"2024-03-04T16:09:11.858799+0000","last_fullsized":"2024-03-04T16:09:11.858799+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:41.311717+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":344,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327721+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:09:08.327721+0000","last_peered":"2024-03-04T16:09:08.327721+0000","last_clean":"2024-03-04T16:09:08.327721+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:09:08.327721+0000","last_undegraded":"2024-03-04T16:09:08.327721+0000","last_fullsized":"2024-03-04T16:09:08.327721+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:37.704629+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323075+0000","last_change":"2024-03-04T16:06:52.428899+0000","last_active":"2024-03-04T16:09:08.323075+0000","last_peered":"2024-03-04T16:09:08.323075+0000","last_clean":"2024-03-04T16:09:08.323075+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:09:08.323075+0000","last_undegraded":"2024-03-04T16:09:08.323075+0000","last_fullsized":"2024-03-04T16:09:08.323075+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:06:52.428782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:06:52.428782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":258,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325854+0000","last_change":"2024-03-04T16:05:27.006917+0000","last_active":"2024-03-04T16:09:08.325854+0000","last_peered":"2024-03-04T16:09:08.325854+0000","last_clean":"2024-03-04T16:09:08.325854+0000","last_became_active":"2024-03-04T16:04:22.731236+0000","last_became_peered":"2024-03-04T16:04:22.731236+0000","last_unstale":"2024-03-04T16:09:08.325854+0000","last_undegraded":"2024-03-04T16:09:08.325854+0000","last_fullsized":"2024-03-04T16:09:08.325854+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:05:27.006656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:05:27.006656+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321530+0000","last_change":"2024-03-04T16:08:27.876805+0000","last_active":"2024-03-04T16:09:08.321530+0000","last_peered":"2024-03-04T16:09:08.321530+0000","last_clean":"2024-03-04T16:09:08.321530+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:09:08.321530+0000","last_undegraded":"2024-03-04T16:09:08.321530+0000","last_fullsized":"2024-03-04T16:09:08.321530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.279216+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":316,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326146+0000","last_change":"2024-03-04T16:08:07.413971+0000","last_active":"2024-03-04T16:09:08.326146+0000","last_peered":"2024-03-04T16:09:08.326146+0000","last_clean":"2024-03-04T16:09:08.326146+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:09:08.326146+0000","last_undegraded":"2024-03-04T16:09:08.326146+0000","last_fullsized":"2024-03-04T16:09:08.326146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:26.316672+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":138,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322840+0000","last_change":"2024-03-04T16:08:57.778070+0000","last_active":"2024-03-04T16:09:08.322840+0000","last_peered":"2024-03-04T16:09:08.322840+0000","last_clean":"2024-03-04T16:09:08.322840+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:09:08.322840+0000","last_undegraded":"2024-03-04T16:09:08.322840+0000","last_fullsized":"2024-03-04T16:09:08.322840+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:57.777958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.777958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.367940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327811+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:09:08.327811+0000","last_peered":"2024-03-04T16:09:08.327811+0000","last_clean":"2024-03-04T16:09:08.327811+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:09:08.327811+0000","last_undegraded":"2024-03-04T16:09:08.327811+0000","last_fullsized":"2024-03-04T16:09:08.327811+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.493416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321712+0000","last_change":"2024-03-04T16:08:12.603101+0000","last_active":"2024-03-04T16:09:08.321712+0000","last_peered":"2024-03-04T16:09:08.321712+0000","last_clean":"2024-03-04T16:09:08.321712+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:09:08.321712+0000","last_undegraded":"2024-03-04T16:09:08.321712+0000","last_fullsized":"2024-03-04T16:09:08.321712+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.602969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.602969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:38.139716+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":229,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326755+0000","last_change":"2024-03-04T16:06:36.235353+0000","last_active":"2024-03-04T16:09:08.326755+0000","last_peered":"2024-03-04T16:09:08.326755+0000","last_clean":"2024-03-04T16:09:08.326755+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:09:08.326755+0000","last_undegraded":"2024-03-04T16:09:08.326755+0000","last_fullsized":"2024-03-04T16:09:08.326755+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:36.235220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:36.235220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322668+0000","last_change":"2024-03-04T16:09:07.827652+0000","last_active":"2024-03-04T16:09:08.322668+0000","last_peered":"2024-03-04T16:09:08.322668+0000","last_clean":"2024-03-04T16:09:08.322668+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:09:08.322668+0000","last_undegraded":"2024-03-04T16:09:08.322668+0000","last_fullsized":"2024-03-04T16:09:08.322668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:11.354299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":91,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330191+0000","last_change":"2024-03-04T16:08:20.028471+0000","last_active":"2024-03-04T16:09:08.330191+0000","last_peered":"2024-03-04T16:09:08.330191+0000","last_clean":"2024-03-04T16:09:08.330191+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:09:08.330191+0000","last_undegraded":"2024-03-04T16:09:08.330191+0000","last_fullsized":"2024-03-04T16:09:08.330191+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:20.025566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.025566+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.552865+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323016+0000","last_change":"2024-03-04T16:08:19.818086+0000","last_active":"2024-03-04T16:09:08.323016+0000","last_peered":"2024-03-04T16:09:08.323016+0000","last_clean":"2024-03-04T16:09:08.323016+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:09:08.323016+0000","last_undegraded":"2024-03-04T16:09:08.323016+0000","last_fullsized":"2024-03-04T16:09:08.323016+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.818005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.818005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:35.179560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326663+0000","last_change":"2024-03-04T16:08:10.136377+0000","last_active":"2024-03-04T16:09:08.326663+0000","last_peered":"2024-03-04T16:09:08.326663+0000","last_clean":"2024-03-04T16:09:08.326663+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:09:08.326663+0000","last_undegraded":"2024-03-04T16:09:08.326663+0000","last_fullsized":"2024-03-04T16:09:08.326663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.136229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.136229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:14.081003+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":265,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325045+0000","last_change":"2024-03-04T16:08:18.893496+0000","last_active":"2024-03-04T16:09:08.325045+0000","last_peered":"2024-03-04T16:09:08.325045+0000","last_clean":"2024-03-04T16:09:08.325045+0000","last_became_active":"2024-03-04T16:07:36.206947+0000","last_became_peered":"2024-03-04T16:07:36.206947+0000","last_unstale":"2024-03-04T16:09:08.325045+0000","last_undegraded":"2024-03-04T16:09:08.325045+0000","last_fullsized":"2024-03-04T16:09:08.325045+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.893370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.893370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.386307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326942+0000","last_change":"2024-03-04T16:06:23.314048+0000","last_active":"2024-03-04T16:09:08.326942+0000","last_peered":"2024-03-04T16:09:08.326942+0000","last_clean":"2024-03-04T16:09:08.326942+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:09:08.326942+0000","last_undegraded":"2024-03-04T16:09:08.326942+0000","last_fullsized":"2024-03-04T16:09:08.326942+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:06:23.313799+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:06:23.313799+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"172'822","reported_seq":842,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.392178+0000","last_change":"2024-03-04T16:08:56.231065+0000","last_active":"2024-03-04T16:09:12.392178+0000","last_peered":"2024-03-04T16:09:12.392178+0000","last_clean":"2024-03-04T16:09:12.392178+0000","last_became_active":"2024-03-04T16:08:56.230573+0000","last_became_peered":"2024-03-04T16:08:56.230573+0000","last_unstale":"2024-03-04T16:09:12.392178+0000","last_undegraded":"2024-03-04T16:09:12.392178+0000","last_fullsized":"2024-03-04T16:09:12.392178+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":822,"log_dups_size":0,"ondisk_log_size":822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.913860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53870592,"num_objects":822,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":822,"num_write_kb":52608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327176+0000","last_change":"2024-03-04T16:08:56.044901+0000","last_active":"2024-03-04T16:09:08.327176+0000","last_peered":"2024-03-04T16:09:08.327176+0000","last_clean":"2024-03-04T16:09:08.327176+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:09:08.327176+0000","last_undegraded":"2024-03-04T16:09:08.327176+0000","last_fullsized":"2024-03-04T16:09:08.327176+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.356711+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":281,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323239+0000","last_change":"2024-03-04T16:08:32.803550+0000","last_active":"2024-03-04T16:09:08.323239+0000","last_peered":"2024-03-04T16:09:08.323239+0000","last_clean":"2024-03-04T16:09:08.323239+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:09:08.323239+0000","last_undegraded":"2024-03-04T16:09:08.323239+0000","last_fullsized":"2024-03-04T16:09:08.323239+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:08:32.803433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:32.803433+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.264477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":273,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326682+0000","last_change":"2024-03-04T16:08:26.222236+0000","last_active":"2024-03-04T16:09:08.326682+0000","last_peered":"2024-03-04T16:09:08.326682+0000","last_clean":"2024-03-04T16:09:08.326682+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:09:08.326682+0000","last_undegraded":"2024-03-04T16:09:08.326682+0000","last_fullsized":"2024-03-04T16:09:08.326682+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.781685+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":323,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322956+0000","last_change":"2024-03-04T16:09:06.869919+0000","last_active":"2024-03-04T16:09:08.322956+0000","last_peered":"2024-03-04T16:09:08.322956+0000","last_clean":"2024-03-04T16:09:08.322956+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:09:08.322956+0000","last_undegraded":"2024-03-04T16:09:08.322956+0000","last_fullsized":"2024-03-04T16:09:08.322956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:06.869771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.869771+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.331005+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"172'858","reported_seq":878,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.415720+0000","last_change":"2024-03-04T16:08:56.789776+0000","last_active":"2024-03-04T16:09:12.415720+0000","last_peered":"2024-03-04T16:09:12.415720+0000","last_clean":"2024-03-04T16:09:12.415720+0000","last_became_active":"2024-03-04T16:08:56.789279+0000","last_became_peered":"2024-03-04T16:08:56.789279+0000","last_unstale":"2024-03-04T16:09:12.415720+0000","last_undegraded":"2024-03-04T16:09:12.415720+0000","last_fullsized":"2024-03-04T16:09:12.415720+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.712859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":858,"num_write_kb":54912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":225,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326489+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:09:08.326489+0000","last_peered":"2024-03-04T16:09:08.326489+0000","last_clean":"2024-03-04T16:09:08.326489+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:09:08.326489+0000","last_undegraded":"2024-03-04T16:09:08.326489+0000","last_fullsized":"2024-03-04T16:09:08.326489+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:22.055486+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323589+0000","last_change":"2024-03-04T16:08:29.793308+0000","last_active":"2024-03-04T16:09:08.323589+0000","last_peered":"2024-03-04T16:09:08.323589+0000","last_clean":"2024-03-04T16:09:08.323589+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:09:08.323589+0000","last_undegraded":"2024-03-04T16:09:08.323589+0000","last_fullsized":"2024-03-04T16:09:08.323589+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:29.793217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.793217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.590776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321580+0000","last_change":"2024-03-04T16:07:24.247724+0000","last_active":"2024-03-04T16:09:08.321580+0000","last_peered":"2024-03-04T16:09:08.321580+0000","last_clean":"2024-03-04T16:09:08.321580+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:09:08.321580+0000","last_undegraded":"2024-03-04T16:09:08.321580+0000","last_fullsized":"2024-03-04T16:09:08.321580+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:07:24.247330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:24.247330+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320901+0000","last_change":"2024-03-04T16:08:03.511267+0000","last_active":"2024-03-04T16:09:08.320901+0000","last_peered":"2024-03-04T16:09:08.320901+0000","last_clean":"2024-03-04T16:09:08.320901+0000","last_became_active":"2024-03-04T16:07:36.213031+0000","last_became_peered":"2024-03-04T16:07:36.213031+0000","last_unstale":"2024-03-04T16:09:08.320901+0000","last_undegraded":"2024-03-04T16:09:08.320901+0000","last_fullsized":"2024-03-04T16:09:08.320901+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:03.511022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:03.511022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:16.777982+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"172'883","reported_seq":903,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.945923+0000","last_change":"2024-03-04T16:08:56.789841+0000","last_active":"2024-03-04T16:09:12.945923+0000","last_peered":"2024-03-04T16:09:12.945923+0000","last_clean":"2024-03-04T16:09:12.945923+0000","last_became_active":"2024-03-04T16:08:56.789562+0000","last_became_peered":"2024-03-04T16:08:56.789562+0000","last_unstale":"2024-03-04T16:09:12.945923+0000","last_undegraded":"2024-03-04T16:09:12.945923+0000","last_fullsized":"2024-03-04T16:09:12.945923+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":883,"log_dups_size":0,"ondisk_log_size":883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.078343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57868288,"num_objects":883,"num_object_clones":0,"num_object_copies":2649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":883,"num_write_kb":56512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.830085+0000","last_change":"2024-03-04T16:09:10.830111+0000","last_active":"2024-03-04T16:09:10.830085+0000","last_peered":"2024-03-04T16:09:10.830085+0000","last_clean":"2024-03-04T16:09:10.830085+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:09:10.830085+0000","last_undegraded":"2024-03-04T16:09:10.830085+0000","last_fullsized":"2024-03-04T16:09:10.830085+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.145188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320782+0000","last_change":"2024-03-04T16:08:22.679372+0000","last_active":"2024-03-04T16:09:08.320782+0000","last_peered":"2024-03-04T16:09:08.320782+0000","last_clean":"2024-03-04T16:09:08.320782+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:09:08.320782+0000","last_undegraded":"2024-03-04T16:09:08.320782+0000","last_fullsized":"2024-03-04T16:09:08.320782+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.679241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.679241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:49.797013+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323370+0000","last_change":"2024-03-04T16:08:23.747781+0000","last_active":"2024-03-04T16:09:08.323370+0000","last_peered":"2024-03-04T16:09:08.323370+0000","last_clean":"2024-03-04T16:09:08.323370+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:09:08.323370+0000","last_undegraded":"2024-03-04T16:09:08.323370+0000","last_fullsized":"2024-03-04T16:09:08.323370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.747649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.747649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:27.232041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":280,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325340+0000","last_change":"2024-03-04T16:08:02.681542+0000","last_active":"2024-03-04T16:09:08.325340+0000","last_peered":"2024-03-04T16:09:08.325340+0000","last_clean":"2024-03-04T16:09:08.325340+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:09:08.325340+0000","last_undegraded":"2024-03-04T16:09:08.325340+0000","last_fullsized":"2024-03-04T16:09:08.325340+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:08:02.681407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.681407+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:22.699986+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"172'875","reported_seq":892,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.709821+0000","last_change":"2024-03-04T16:08:56.234782+0000","last_active":"2024-03-04T16:09:12.709821+0000","last_peered":"2024-03-04T16:09:12.709821+0000","last_clean":"2024-03-04T16:09:12.709821+0000","last_became_active":"2024-03-04T16:08:56.234486+0000","last_became_peered":"2024-03-04T16:08:56.234486+0000","last_unstale":"2024-03-04T16:09:12.709821+0000","last_undegraded":"2024-03-04T16:09:12.709821+0000","last_fullsized":"2024-03-04T16:09:12.709821+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.872524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":56000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":106,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322428+0000","last_change":"2024-03-04T16:07:36.212307+0000","last_active":"2024-03-04T16:09:08.322428+0000","last_peered":"2024-03-04T16:09:08.322428+0000","last_clean":"2024-03-04T16:09:08.322428+0000","last_became_active":"2024-03-04T16:07:36.211991+0000","last_became_peered":"2024-03-04T16:07:36.211991+0000","last_unstale":"2024-03-04T16:09:08.322428+0000","last_undegraded":"2024-03-04T16:09:08.322428+0000","last_fullsized":"2024-03-04T16:09:08.322428+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:06:47.054678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320758+0000","last_change":"2024-03-04T16:08:20.664979+0000","last_active":"2024-03-04T16:09:08.320758+0000","last_peered":"2024-03-04T16:09:08.320758+0000","last_clean":"2024-03-04T16:09:08.320758+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:09:08.320758+0000","last_undegraded":"2024-03-04T16:09:08.320758+0000","last_fullsized":"2024-03-04T16:09:08.320758+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.664848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.403776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327378+0000","last_change":"2024-03-04T16:09:04.159447+0000","last_active":"2024-03-04T16:09:08.327378+0000","last_peered":"2024-03-04T16:09:08.327378+0000","last_clean":"2024-03-04T16:09:08.327378+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:09:08.327378+0000","last_undegraded":"2024-03-04T16:09:08.327378+0000","last_fullsized":"2024-03-04T16:09:08.327378+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.732534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":91,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328489+0000","last_change":"2024-03-04T16:08:22.032175+0000","last_active":"2024-03-04T16:09:08.328489+0000","last_peered":"2024-03-04T16:09:08.328489+0000","last_clean":"2024-03-04T16:09:08.328489+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:09:08.328489+0000","last_undegraded":"2024-03-04T16:09:08.328489+0000","last_fullsized":"2024-03-04T16:09:08.328489+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:08:22.032049+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.032049+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:45.673430+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"172'864","reported_seq":884,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.397619+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:09:12.397619+0000","last_peered":"2024-03-04T16:09:12.397619+0000","last_clean":"2024-03-04T16:09:12.397619+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:09:12.397619+0000","last_undegraded":"2024-03-04T16:09:12.397619+0000","last_fullsized":"2024-03-04T16:09:12.397619+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.332956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":864,"num_write_kb":55296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":225,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326538+0000","last_change":"2024-03-04T16:06:42.215087+0000","last_active":"2024-03-04T16:09:08.326538+0000","last_peered":"2024-03-04T16:09:08.326538+0000","last_clean":"2024-03-04T16:09:08.326538+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:09:08.326538+0000","last_undegraded":"2024-03-04T16:09:08.326538+0000","last_fullsized":"2024-03-04T16:09:08.326538+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:42.214971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:42.214971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327078+0000","last_change":"2024-03-04T16:09:01.160778+0000","last_active":"2024-03-04T16:09:08.327078+0000","last_peered":"2024-03-04T16:09:08.327078+0000","last_clean":"2024-03-04T16:09:08.327078+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:09:08.327078+0000","last_undegraded":"2024-03-04T16:09:08.327078+0000","last_fullsized":"2024-03-04T16:09:08.327078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.894027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":266,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321282+0000","last_change":"2024-03-04T16:08:58.764898+0000","last_active":"2024-03-04T16:09:08.321282+0000","last_peered":"2024-03-04T16:09:08.321282+0000","last_clean":"2024-03-04T16:09:08.321282+0000","last_became_active":"2024-03-04T16:07:36.213232+0000","last_became_peered":"2024-03-04T16:07:36.213232+0000","last_unstale":"2024-03-04T16:09:08.321282+0000","last_undegraded":"2024-03-04T16:09:08.321282+0000","last_fullsized":"2024-03-04T16:09:08.321282+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.604580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":368,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328486+0000","last_change":"2024-03-04T16:08:21.819036+0000","last_active":"2024-03-04T16:09:08.328486+0000","last_peered":"2024-03-04T16:09:08.328486+0000","last_clean":"2024-03-04T16:09:08.328486+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:09:08.328486+0000","last_undegraded":"2024-03-04T16:09:08.328486+0000","last_fullsized":"2024-03-04T16:09:08.328486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.713364+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.329878+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:09:08.329878+0000","last_peered":"2024-03-04T16:09:08.329878+0000","last_clean":"2024-03-04T16:09:08.329878+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:09:08.329878+0000","last_undegraded":"2024-03-04T16:09:08.329878+0000","last_fullsized":"2024-03-04T16:09:08.329878+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:59.260044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"172'903","reported_seq":923,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.910396+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:09:12.910396+0000","last_peered":"2024-03-04T16:09:12.910396+0000","last_clean":"2024-03-04T16:09:12.910396+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:09:12.910396+0000","last_undegraded":"2024-03-04T16:09:12.910396+0000","last_fullsized":"2024-03-04T16:09:12.910396+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":903,"log_dups_size":0,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.644896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59179008,"num_objects":903,"num_object_clones":0,"num_object_copies":2709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":903,"num_write_kb":57792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323500+0000","last_change":"2024-03-04T16:09:03.867404+0000","last_active":"2024-03-04T16:09:08.323500+0000","last_peered":"2024-03-04T16:09:08.323500+0000","last_clean":"2024-03-04T16:09:08.323500+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:09:08.323500+0000","last_undegraded":"2024-03-04T16:09:08.323500+0000","last_fullsized":"2024-03-04T16:09:08.323500+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.867298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.867298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.903366+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330794+0000","last_change":"2024-03-04T16:08:02.744435+0000","last_active":"2024-03-04T16:09:08.330794+0000","last_peered":"2024-03-04T16:09:08.330794+0000","last_clean":"2024-03-04T16:09:08.330794+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:09:08.330794+0000","last_undegraded":"2024-03-04T16:09:08.330794+0000","last_fullsized":"2024-03-04T16:09:08.330794+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:02.744314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.744314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:19.782345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320716+0000","last_change":"2024-03-04T16:08:08.562442+0000","last_active":"2024-03-04T16:09:08.320716+0000","last_peered":"2024-03-04T16:09:08.320716+0000","last_clean":"2024-03-04T16:09:08.320716+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:09:08.320716+0000","last_undegraded":"2024-03-04T16:09:08.320716+0000","last_fullsized":"2024-03-04T16:09:08.320716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:08.562332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:08:08.562332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:20.956438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":309,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330814+0000","last_change":"2024-03-04T16:08:26.885868+0000","last_active":"2024-03-04T16:09:08.330814+0000","last_peered":"2024-03-04T16:09:08.330814+0000","last_clean":"2024-03-04T16:09:08.330814+0000","last_became_active":"2024-03-04T16:07:36.224375+0000","last_became_peered":"2024-03-04T16:07:36.224375+0000","last_unstale":"2024-03-04T16:09:08.330814+0000","last_undegraded":"2024-03-04T16:09:08.330814+0000","last_fullsized":"2024-03-04T16:09:08.330814+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:47.549945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":237,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322477+0000","last_change":"2024-03-04T16:08:10.648202+0000","last_active":"2024-03-04T16:09:08.322477+0000","last_peered":"2024-03-04T16:09:08.322477+0000","last_clean":"2024-03-04T16:09:08.322477+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:09:08.322477+0000","last_undegraded":"2024-03-04T16:09:08.322477+0000","last_fullsized":"2024-03-04T16:09:08.322477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.648089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.648089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:31.126923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"172'799","reported_seq":819,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.400870+0000","last_change":"2024-03-04T16:08:56.231210+0000","last_active":"2024-03-04T16:09:12.400870+0000","last_peered":"2024-03-04T16:09:12.400870+0000","last_clean":"2024-03-04T16:09:12.400870+0000","last_became_active":"2024-03-04T16:08:56.230722+0000","last_became_peered":"2024-03-04T16:08:56.230722+0000","last_unstale":"2024-03-04T16:09:12.400870+0000","last_undegraded":"2024-03-04T16:09:12.400870+0000","last_fullsized":"2024-03-04T16:09:12.400870+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.131015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363264,"num_objects":799,"num_object_clones":0,"num_object_copies":2397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":799,"num_write_kb":51136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326795+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:09:08.326795+0000","last_peered":"2024-03-04T16:09:08.326795+0000","last_clean":"2024-03-04T16:09:08.326795+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:09:08.326795+0000","last_undegraded":"2024-03-04T16:09:08.326795+0000","last_fullsized":"2024-03-04T16:09:08.326795+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.705424+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":528,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.322429+0000","last_change":"2024-03-04T16:07:39.283680+0000","last_active":"2024-03-04T16:09:08.322429+0000","last_peered":"2024-03-04T16:09:08.322429+0000","last_clean":"2024-03-04T16:09:08.322429+0000","last_became_active":"2024-03-04T16:07:39.283273+0000","last_became_peered":"2024-03-04T16:07:39.283273+0000","last_unstale":"2024-03-04T16:09:08.322429+0000","last_undegraded":"2024-03-04T16:09:08.322429+0000","last_fullsized":"2024-03-04T16:09:08.322429+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:06:45.435669+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:06:45.435669+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.324538+0000","last_change":"2024-03-04T16:05:48.401400+0000","last_active":"2024-03-04T16:09:08.324538+0000","last_peered":"2024-03-04T16:09:08.324538+0000","last_clean":"2024-03-04T16:09:08.324538+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:09:08.324538+0000","last_undegraded":"2024-03-04T16:09:08.324538+0000","last_fullsized":"2024-03-04T16:09:08.324538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:05:48.401259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:05:48.401259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":350,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326180+0000","last_change":"2024-03-04T16:08:29.900384+0000","last_active":"2024-03-04T16:09:08.326180+0000","last_peered":"2024-03-04T16:09:08.326180+0000","last_clean":"2024-03-04T16:09:08.326180+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:09:08.326180+0000","last_undegraded":"2024-03-04T16:09:08.326180+0000","last_fullsized":"2024-03-04T16:09:08.326180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:08:29.900265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.900265+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:38.478978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.323011+0000","last_change":"2024-03-04T16:08:21.802175+0000","last_active":"2024-03-04T16:09:08.323011+0000","last_peered":"2024-03-04T16:09:08.323011+0000","last_clean":"2024-03-04T16:09:08.323011+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:09:08.323011+0000","last_undegraded":"2024-03-04T16:09:08.323011+0000","last_fullsized":"2024-03-04T16:09:08.323011+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:21.802092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.802092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:34.204356+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"172'861","reported_seq":881,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.694636+0000","last_change":"2024-03-04T16:08:56.234639+0000","last_active":"2024-03-04T16:09:12.694636+0000","last_peered":"2024-03-04T16:09:12.694636+0000","last_clean":"2024-03-04T16:09:12.694636+0000","last_became_active":"2024-03-04T16:08:56.234353+0000","last_became_peered":"2024-03-04T16:08:56.234353+0000","last_unstale":"2024-03-04T16:09:12.694636+0000","last_undegraded":"2024-03-04T16:09:12.694636+0000","last_fullsized":"2024-03-04T16:09:12.694636+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:05.306737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56426496,"num_objects":861,"num_object_clones":0,"num_object_copies":2583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":861,"num_write_kb":55104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":242,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328119+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:09:08.328119+0000","last_peered":"2024-03-04T16:09:08.328119+0000","last_clean":"2024-03-04T16:09:08.328119+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:09:08.328119+0000","last_undegraded":"2024-03-04T16:09:08.328119+0000","last_fullsized":"2024-03-04T16:09:08.328119+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.688148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":81,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.329098+0000","last_change":"2024-03-04T16:08:10.110344+0000","last_active":"2024-03-04T16:09:08.329098+0000","last_peered":"2024-03-04T16:09:08.329098+0000","last_clean":"2024-03-04T16:09:08.329098+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:09:08.329098+0000","last_undegraded":"2024-03-04T16:09:08.329098+0000","last_fullsized":"2024-03-04T16:09:08.329098+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.109986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.109986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:34.761279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":640,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:10.556892+0000","last_change":"2024-03-04T16:08:03.849126+0000","last_active":"2024-03-04T16:09:10.556892+0000","last_peered":"2024-03-04T16:09:10.556892+0000","last_clean":"2024-03-04T16:09:10.556892+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:09:10.556892+0000","last_undegraded":"2024-03-04T16:09:10.556892+0000","last_fullsized":"2024-03-04T16:09:10.556892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-04T16:08:03.848663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:08:03.848663+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:18.982880+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":377,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":95,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.329944+0000","last_change":"2024-03-04T16:08:16.126230+0000","last_active":"2024-03-04T16:09:08.329944+0000","last_peered":"2024-03-04T16:09:08.329944+0000","last_clean":"2024-03-04T16:09:08.329944+0000","last_became_active":"2024-03-04T16:07:36.231501+0000","last_became_peered":"2024-03-04T16:07:36.231501+0000","last_unstale":"2024-03-04T16:09:08.329944+0000","last_undegraded":"2024-03-04T16:09:08.329944+0000","last_fullsized":"2024-03-04T16:09:08.329944+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:16.125960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:16.125960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:27.499671+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326991+0000","last_change":"2024-03-04T16:08:28.262306+0000","last_active":"2024-03-04T16:09:08.326991+0000","last_peered":"2024-03-04T16:09:08.326991+0000","last_clean":"2024-03-04T16:09:08.326991+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:09:08.326991+0000","last_undegraded":"2024-03-04T16:09:08.326991+0000","last_fullsized":"2024-03-04T16:09:08.326991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.262199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.262199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:54.557979+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"172'864","reported_seq":884,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.874184+0000","last_change":"2024-03-04T16:08:56.233179+0000","last_active":"2024-03-04T16:09:12.874184+0000","last_peered":"2024-03-04T16:09:12.874184+0000","last_clean":"2024-03-04T16:09:12.874184+0000","last_became_active":"2024-03-04T16:08:56.231149+0000","last_became_peered":"2024-03-04T16:08:56.231149+0000","last_unstale":"2024-03-04T16:09:12.874184+0000","last_undegraded":"2024-03-04T16:09:12.874184+0000","last_fullsized":"2024-03-04T16:09:12.874184+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.112092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56623104,"num_objects":864,"num_object_clones":0,"num_object_copies":2592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":864,"num_write_kb":55296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328774+0000","last_change":"2024-03-04T16:08:19.795646+0000","last_active":"2024-03-04T16:09:08.328774+0000","last_peered":"2024-03-04T16:09:08.328774+0000","last_clean":"2024-03-04T16:09:08.328774+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:09:08.328774+0000","last_undegraded":"2024-03-04T16:09:08.328774+0000","last_fullsized":"2024-03-04T16:09:08.328774+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.795237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.795237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:23.809312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328353+0000","last_change":"2024-03-04T16:08:24.836558+0000","last_active":"2024-03-04T16:09:08.328353+0000","last_peered":"2024-03-04T16:09:08.328353+0000","last_clean":"2024-03-04T16:09:08.328353+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:09:08.328353+0000","last_undegraded":"2024-03-04T16:09:08.328353+0000","last_fullsized":"2024-03-04T16:09:08.328353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:50.980312+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":213,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325790+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:09:08.325790+0000","last_peered":"2024-03-04T16:09:08.325790+0000","last_clean":"2024-03-04T16:09:08.325790+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:09:08.325790+0000","last_undegraded":"2024-03-04T16:09:08.325790+0000","last_fullsized":"2024-03-04T16:09:08.325790+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:48.807304+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325348+0000","last_change":"2024-03-04T16:08:06.669836+0000","last_active":"2024-03-04T16:09:08.325348+0000","last_peered":"2024-03-04T16:09:08.325348+0000","last_clean":"2024-03-04T16:09:08.325348+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:09:08.325348+0000","last_undegraded":"2024-03-04T16:09:08.325348+0000","last_fullsized":"2024-03-04T16:09:08.325348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.669723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.669723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:25.923627+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"172'921","reported_seq":941,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.943041+0000","last_change":"2024-03-04T16:08:56.229590+0000","last_active":"2024-03-04T16:09:12.943041+0000","last_peered":"2024-03-04T16:09:12.943041+0000","last_clean":"2024-03-04T16:09:12.943041+0000","last_became_active":"2024-03-04T16:08:56.229291+0000","last_became_peered":"2024-03-04T16:08:56.229291+0000","last_unstale":"2024-03-04T16:09:12.943041+0000","last_undegraded":"2024-03-04T16:09:12.943041+0000","last_fullsized":"2024-03-04T16:09:12.943041+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:04.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60358656,"num_objects":921,"num_object_clones":0,"num_object_copies":2763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":921,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":921,"num_write_kb":58944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328281+0000","last_change":"2024-03-04T16:07:35.972227+0000","last_active":"2024-03-04T16:09:08.328281+0000","last_peered":"2024-03-04T16:09:08.328281+0000","last_clean":"2024-03-04T16:09:08.328281+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:09:08.328281+0000","last_undegraded":"2024-03-04T16:09:08.328281+0000","last_fullsized":"2024-03-04T16:09:08.328281+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:35.972109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:07:35.972109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":253,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.013448+0000","last_change":"2024-03-04T16:09:12.013484+0000","last_active":"2024-03-04T16:09:12.013448+0000","last_peered":"2024-03-04T16:09:12.013448+0000","last_clean":"2024-03-04T16:09:12.013448+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:09:12.013448+0000","last_undegraded":"2024-03-04T16:09:12.013448+0000","last_fullsized":"2024-03-04T16:09:12.013448+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.570727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":262,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330425+0000","last_change":"2024-03-04T16:08:17.809580+0000","last_active":"2024-03-04T16:09:08.330425+0000","last_peered":"2024-03-04T16:09:08.330425+0000","last_clean":"2024-03-04T16:09:08.330425+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:09:08.330425+0000","last_undegraded":"2024-03-04T16:09:08.330425+0000","last_fullsized":"2024-03-04T16:09:08.330425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:08:17.809443+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.809443+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:19.479582+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"141'10","reported_seq":175,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.838490+0000","last_change":"2024-03-04T16:08:13.854540+0000","last_active":"2024-03-04T16:09:08.838490+0000","last_peered":"2024-03-04T16:09:08.838490+0000","last_clean":"2024-03-04T16:09:08.838490+0000","last_became_active":"2024-03-04T16:07:36.219930+0000","last_became_peered":"2024-03-04T16:07:36.219930+0000","last_unstale":"2024-03-04T16:09:08.838490+0000","last_undegraded":"2024-03-04T16:09:08.838490+0000","last_fullsized":"2024-03-04T16:09:08.838490+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:13.854454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.854454+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.923746+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"172'801","reported_seq":821,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.403692+0000","last_change":"2024-03-04T16:08:56.789726+0000","last_active":"2024-03-04T16:09:12.403692+0000","last_peered":"2024-03-04T16:09:12.403692+0000","last_clean":"2024-03-04T16:09:12.403692+0000","last_became_active":"2024-03-04T16:08:56.789091+0000","last_became_peered":"2024-03-04T16:08:56.789091+0000","last_unstale":"2024-03-04T16:09:12.403692+0000","last_undegraded":"2024-03-04T16:09:12.403692+0000","last_fullsized":"2024-03-04T16:09:12.403692+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.511629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":801,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":51264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.324072+0000","last_change":"2024-03-04T16:06:49.407403+0000","last_active":"2024-03-04T16:09:08.324072+0000","last_peered":"2024-03-04T16:09:08.324072+0000","last_clean":"2024-03-04T16:09:08.324072+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:09:08.324072+0000","last_undegraded":"2024-03-04T16:09:08.324072+0000","last_fullsized":"2024-03-04T16:09:08.324072+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:06:49.407279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":85,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330074+0000","last_change":"2024-03-04T16:08:17.085244+0000","last_active":"2024-03-04T16:09:08.330074+0000","last_peered":"2024-03-04T16:09:08.330074+0000","last_clean":"2024-03-04T16:09:08.330074+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:09:08.330074+0000","last_undegraded":"2024-03-04T16:09:08.330074+0000","last_fullsized":"2024-03-04T16:09:08.330074+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:17.084994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.084994+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:19.135434+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":270,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321265+0000","last_change":"2024-03-04T16:08:59.783945+0000","last_active":"2024-03-04T16:09:08.321265+0000","last_peered":"2024-03-04T16:09:08.321265+0000","last_clean":"2024-03-04T16:09:08.321265+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:09:08.321265+0000","last_undegraded":"2024-03-04T16:09:08.321265+0000","last_fullsized":"2024-03-04T16:09:08.321265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:00.650777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":316,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326084+0000","last_change":"2024-03-04T16:06:25.283072+0000","last_active":"2024-03-04T16:09:08.326084+0000","last_peered":"2024-03-04T16:09:08.326084+0000","last_clean":"2024-03-04T16:09:08.326084+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:09:08.326084+0000","last_undegraded":"2024-03-04T16:09:08.326084+0000","last_fullsized":"2024-03-04T16:09:08.326084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:06:25.282843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:25.282843+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"172'953","reported_seq":973,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.891843+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:09:12.891843+0000","last_peered":"2024-03-04T16:09:12.891843+0000","last_clean":"2024-03-04T16:09:12.891843+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:09:12.891843+0000","last_undegraded":"2024-03-04T16:09:12.891843+0000","last_fullsized":"2024-03-04T16:09:12.891843+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.412376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62455808,"num_objects":953,"num_object_clones":0,"num_object_copies":2859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":953,"num_write_kb":60992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320911+0000","last_change":"2024-03-04T16:09:05.803480+0000","last_active":"2024-03-04T16:09:08.320911+0000","last_peered":"2024-03-04T16:09:08.320911+0000","last_clean":"2024-03-04T16:09:08.320911+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:09:08.320911+0000","last_undegraded":"2024-03-04T16:09:08.320911+0000","last_fullsized":"2024-03-04T16:09:08.320911+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:30.872236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326585+0000","last_change":"2024-03-04T16:07:36.221669+0000","last_active":"2024-03-04T16:09:08.326585+0000","last_peered":"2024-03-04T16:09:08.326585+0000","last_clean":"2024-03-04T16:09:08.326585+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:09:08.326585+0000","last_undegraded":"2024-03-04T16:09:08.326585+0000","last_fullsized":"2024-03-04T16:09:08.326585+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:06:49.140391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":322,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325806+0000","last_change":"2024-03-04T16:08:02.407502+0000","last_active":"2024-03-04T16:09:08.325806+0000","last_peered":"2024-03-04T16:09:08.325806+0000","last_clean":"2024-03-04T16:09:08.325806+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:09:08.325806+0000","last_undegraded":"2024-03-04T16:09:08.325806+0000","last_fullsized":"2024-03-04T16:09:08.325806+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:08:02.407290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.407290+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:18.961185+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320951+0000","last_change":"2024-03-04T16:07:28.229798+0000","last_active":"2024-03-04T16:09:08.320951+0000","last_peered":"2024-03-04T16:09:08.320951+0000","last_clean":"2024-03-04T16:09:08.320951+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:09:08.320951+0000","last_undegraded":"2024-03-04T16:09:08.320951+0000","last_fullsized":"2024-03-04T16:09:08.320951+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:28.229716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:07:28.229716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"172'849","reported_seq":869,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.709660+0000","last_change":"2024-03-04T16:08:56.232940+0000","last_active":"2024-03-04T16:09:12.709660+0000","last_peered":"2024-03-04T16:09:12.709660+0000","last_clean":"2024-03-04T16:09:12.709660+0000","last_became_active":"2024-03-04T16:08:56.232588+0000","last_became_peered":"2024-03-04T16:08:56.232588+0000","last_unstale":"2024-03-04T16:09:12.709660+0000","last_undegraded":"2024-03-04T16:09:12.709660+0000","last_fullsized":"2024-03-04T16:09:12.709660+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.367142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55640064,"num_objects":849,"num_object_clones":0,"num_object_copies":2547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":849,"num_write_kb":54336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.327263+0000","last_change":"2024-03-04T16:08:17.162298+0000","last_active":"2024-03-04T16:09:08.327263+0000","last_peered":"2024-03-04T16:09:08.327263+0000","last_clean":"2024-03-04T16:09:08.327263+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:09:08.327263+0000","last_undegraded":"2024-03-04T16:09:08.327263+0000","last_fullsized":"2024-03-04T16:09:08.327263+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:17.162173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.162173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.157003+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":323,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321266+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:09:08.321266+0000","last_peered":"2024-03-04T16:09:08.321266+0000","last_clean":"2024-03-04T16:09:08.321266+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:09:08.321266+0000","last_undegraded":"2024-03-04T16:09:08.321266+0000","last_fullsized":"2024-03-04T16:09:08.321266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:29.077001+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325731+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:09:08.325731+0000","last_peered":"2024-03-04T16:09:08.325731+0000","last_clean":"2024-03-04T16:09:08.325731+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:09:08.325731+0000","last_undegraded":"2024-03-04T16:09:08.325731+0000","last_fullsized":"2024-03-04T16:09:08.325731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.366856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":266,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.330308+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:09:08.330308+0000","last_peered":"2024-03-04T16:09:08.330308+0000","last_clean":"2024-03-04T16:09:08.330308+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:09:08.330308+0000","last_undegraded":"2024-03-04T16:09:08.330308+0000","last_fullsized":"2024-03-04T16:09:08.330308+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:37.375921+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"172'858","reported_seq":878,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.364739+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:09:12.364739+0000","last_peered":"2024-03-04T16:09:12.364739+0000","last_clean":"2024-03-04T16:09:12.364739+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:09:12.364739+0000","last_undegraded":"2024-03-04T16:09:12.364739+0000","last_fullsized":"2024-03-04T16:09:12.364739+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56229888,"num_objects":858,"num_object_clones":0,"num_object_copies":2574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":858,"num_write_kb":54912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":226,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325730+0000","last_change":"2024-03-04T16:08:12.467279+0000","last_active":"2024-03-04T16:09:08.325730+0000","last_peered":"2024-03-04T16:09:08.325730+0000","last_clean":"2024-03-04T16:09:08.325730+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:09:08.325730+0000","last_undegraded":"2024-03-04T16:09:08.325730+0000","last_fullsized":"2024-03-04T16:09:08.325730+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.082303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":244,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.326986+0000","last_change":"2024-03-04T16:09:06.138702+0000","last_active":"2024-03-04T16:09:08.326986+0000","last_peered":"2024-03-04T16:09:08.326986+0000","last_clean":"2024-03-04T16:09:08.326986+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:09:08.326986+0000","last_undegraded":"2024-03-04T16:09:08.326986+0000","last_fullsized":"2024-03-04T16:09:08.326986+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.908387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":246,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321139+0000","last_change":"2024-03-04T16:09:06.758717+0000","last_active":"2024-03-04T16:09:08.321139+0000","last_peered":"2024-03-04T16:09:08.321139+0000","last_clean":"2024-03-04T16:09:08.321139+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:09:08.321139+0000","last_undegraded":"2024-03-04T16:09:08.321139+0000","last_fullsized":"2024-03-04T16:09:08.321139+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:27.733168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":265,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321080+0000","last_change":"2024-03-04T16:08:02.022242+0000","last_active":"2024-03-04T16:09:08.321080+0000","last_peered":"2024-03-04T16:09:08.321080+0000","last_clean":"2024-03-04T16:09:08.321080+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:09:08.321080+0000","last_undegraded":"2024-03-04T16:09:08.321080+0000","last_fullsized":"2024-03-04T16:09:08.321080+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:08:02.022098+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.022098+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:15.537249+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"172'869","reported_seq":888,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.705372+0000","last_change":"2024-03-04T16:08:56.233870+0000","last_active":"2024-03-04T16:09:12.705372+0000","last_peered":"2024-03-04T16:09:12.705372+0000","last_clean":"2024-03-04T16:09:12.705372+0000","last_became_active":"2024-03-04T16:08:56.233555+0000","last_became_peered":"2024-03-04T16:08:56.233555+0000","last_unstale":"2024-03-04T16:09:12.705372+0000","last_undegraded":"2024-03-04T16:09:12.705372+0000","last_fullsized":"2024-03-04T16:09:12.705372+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":869,"log_dups_size":0,"ondisk_log_size":869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.436373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950784,"num_objects":869,"num_object_clones":0,"num_object_copies":2607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":55616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.320843+0000","last_change":"2024-03-04T16:08:30.840092+0000","last_active":"2024-03-04T16:09:08.320843+0000","last_peered":"2024-03-04T16:09:08.320843+0000","last_clean":"2024-03-04T16:09:08.320843+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:09:08.320843+0000","last_undegraded":"2024-03-04T16:09:08.320843+0000","last_fullsized":"2024-03-04T16:09:08.320843+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.142723+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.328541+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:09:08.328541+0000","last_peered":"2024-03-04T16:09:08.328541+0000","last_clean":"2024-03-04T16:09:08.328541+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:09:08.328541+0000","last_undegraded":"2024-03-04T16:09:08.328541+0000","last_fullsized":"2024-03-04T16:09:08.328541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:28.310699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":245,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325466+0000","last_change":"2024-03-04T16:08:07.718705+0000","last_active":"2024-03-04T16:09:08.325466+0000","last_peered":"2024-03-04T16:09:08.325466+0000","last_clean":"2024-03-04T16:09:08.325466+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:09:08.325466+0000","last_undegraded":"2024-03-04T16:09:08.325466+0000","last_fullsized":"2024-03-04T16:09:08.325466+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:07.718581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.718581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:20.685302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":269,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321508+0000","last_change":"2024-03-04T16:07:22.316627+0000","last_active":"2024-03-04T16:09:08.321508+0000","last_peered":"2024-03-04T16:09:08.321508+0000","last_clean":"2024-03-04T16:09:08.321508+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:09:08.321508+0000","last_undegraded":"2024-03-04T16:09:08.321508+0000","last_fullsized":"2024-03-04T16:09:08.321508+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:07:22.316461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:22.316461+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"172'888","reported_seq":908,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:12.903641+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:09:12.903641+0000","last_peered":"2024-03-04T16:09:12.903641+0000","last_clean":"2024-03-04T16:09:12.903641+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:09:12.903641+0000","last_undegraded":"2024-03-04T16:09:12.903641+0000","last_fullsized":"2024-03-04T16:09:12.903641+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:09.458186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":888,"num_write_kb":56832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321311+0000","last_change":"2024-03-04T16:08:13.787566+0000","last_active":"2024-03-04T16:09:08.321311+0000","last_peered":"2024-03-04T16:09:08.321311+0000","last_clean":"2024-03-04T16:09:08.321311+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:09:08.321311+0000","last_undegraded":"2024-03-04T16:09:08.321311+0000","last_fullsized":"2024-03-04T16:09:08.321311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:29.686423+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":335,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.321148+0000","last_change":"2024-03-04T16:08:28.762450+0000","last_active":"2024-03-04T16:09:08.321148+0000","last_peered":"2024-03-04T16:09:08.321148+0000","last_clean":"2024-03-04T16:09:08.321148+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:09:08.321148+0000","last_undegraded":"2024-03-04T16:09:08.321148+0000","last_fullsized":"2024-03-04T16:09:08.321148+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-04T16:08:28.762324+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.762324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.916266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":249,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.325797+0000","last_change":"2024-03-04T16:07:36.224345+0000","last_active":"2024-03-04T16:09:08.325797+0000","last_peered":"2024-03-04T16:09:08.325797+0000","last_clean":"2024-03-04T16:09:08.325797+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:09:08.325797+0000","last_undegraded":"2024-03-04T16:09:08.325797+0000","last_fullsized":"2024-03-04T16:09:08.325797+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:51.144836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:06:51.144836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-03-04T16:09:08.714716+0000","last_change":"2024-03-04T16:09:08.714773+0000","last_active":"2024-03-04T16:09:08.714716+0000","last_peered":"2024-03-04T16:09:08.714716+0000","last_clean":"2024-03-04T16:09:08.714716+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:09:08.714716+0000","last_undegraded":"2024-03-04T16:09:08.714716+0000","last_fullsized":"2024-03-04T16:09:08.714716+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.690427+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":908853248,"num_objects":13868,"num_object_clones":0,"num_object_copies":41604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13868,"num_write_kb":887552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2698838016,"data_stored":2698838016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13868,"ondisk_log_size":13868,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":382,"num_read_kb":377,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070546,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390940,"kb_used_data":243156,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93354212,"statfs":{"total":95995035648,"available":95594713088,"internally_reserved":0,"allocated":248991744,"data_stored":248569338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364886,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":382020,"kb_used_data":248124,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93363132,"statfs":{"total":95995035648,"available":95603847168,"internally_reserved":0,"allocated":254078976,"data_stored":253678834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":137099718},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":586100,"kb_used_data":384172,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93159052,"statfs":{"total":95995035648,"available":95394869248,"internally_reserved":0,"allocated":393392128,"data_stored":392943842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477852,"kb_used_data":316180,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93267300,"statfs":{"total":95995035648,"available":95505715200,"internally_reserved":0,"allocated":323768320,"data_stored":323345258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":428036,"kb_used_data":288572,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93317116,"statfs":{"total":95995035648,"available":95556726784,"internally_reserved":0,"allocated":295497728,"data_stored":295097762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529320,"kb_used_data":291296,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93215832,"statfs":{"total":95995035648,"available":95453011968,"internally_reserved":0,"allocated":298287104,"data_stored":297851320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":243726792},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355612,"kb_used_data":234196,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93389540,"statfs":{"total":95995035648,"available":95630888960,"internally_reserved":0,"allocated":239816704,"data_stored":239392432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626524,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":532356,"kb_used_data":392828,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93212796,"statfs":{"total":95995035648,"available":95449903104,"internally_reserved":0,"allocated":402255872,"data_stored":401855512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142801350},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":357236736,"data_stored":357236736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":394395648,"data_stored":394395648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":332857344,"data_stored":332857344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":355532800,"data_stored":355532800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":276299776,"data_stored":276299776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":432472064,"data_stored":432472064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":271319040,"data_stored":271319040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":278724608,"data_stored":278724608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:09:22.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:09:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:22.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:22.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:22.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:22.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:22.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:22.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:22.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:22.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:09:22.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[29779]: pgmap v275: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-04T16:09:22.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:22.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:22.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:09:22.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[38113]: pgmap v275: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-04T16:09:22.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:22.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.596 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:22.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:22.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:22.738 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:22.795 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:09:22.795 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:09:22.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:22 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:09:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:22 smithi176 ceph-mon[33619]: pgmap v275: 148 pgs: 148 active+clean; 1.5 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-04T16:09:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:22.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:22.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:22.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:22.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:23.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:23.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 15 15 35168 35153 146.447 111 0.00649625 0.00682309 2024-03-04T16:09:23.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:23.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:23.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:23.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:23.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:23.398 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:23.411 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:23.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 2.0 deep-scrub starts 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 2.0 deep-scrub ok 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 6.1a deep-scrub starts 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 6.1a deep-scrub ok 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:09:23.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 2.0 deep-scrub starts 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 2.0 deep-scrub ok 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 6.1a deep-scrub starts 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 6.1a deep-scrub ok 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:09:23.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:23 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:09:23.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:23.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:23.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:23.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:23.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:23.820 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 2.0 deep-scrub starts 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 2.0 deep-scrub ok 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:09:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:09:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 6.1a deep-scrub starts 2024-03-04T16:09:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 6.1a deep-scrub ok 2024-03-04T16:09:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:09:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:23 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:09:23.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:23.959 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:23.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:24.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:24.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 16 16 37031 37015 144.567 116.375 0.00591233 0.00691213 2024-03-04T16:09:24.094 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:24.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:24.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:24.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:24.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:24.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:24.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:24.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:24.517 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:09:24.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:24.648 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: pgmap v276: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/123814641' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: from='client.25051 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: pgmap v276: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/123814641' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: from='client.25051 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:09:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:24 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:09:24.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:24.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:24.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:09:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:09:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:09:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: pgmap v276: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/123814641' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: from='client.25051 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:09:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:24 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:09:24.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:24.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:24.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:25.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:25.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 17 16 38952 38936 143.125 120.062 0.0155701 0.00698059 2024-03-04T16:09:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:25.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:25.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:25.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:25.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:25.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:25.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:25.519 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:25.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:25.662 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[29779]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[29779]: osdmap e173: 8 total, 8 up, 8 in 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:09:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:09:25.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:25.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[38113]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:09:25.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:25 smithi038 ceph-mon[38113]: osdmap e173: 8 total, 8 up, 8 in 2024-03-04T16:09:25.764 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:25.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:25 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:09:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:25 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:09:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:25 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:25 smithi176 ceph-mon[33619]: from='client.25051 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:09:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:25 smithi176 ceph-mon[33619]: osdmap e173: 8 total, 8 up, 8 in 2024-03-04T16:09:25.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:25.935 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:25.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:26.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:26.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 18 16 40830 40814 141.694 117.375 0.0109284 0.007051 2024-03-04T16:09:26.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:26.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:26.218 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:26.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:26.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:26.355 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:26.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:26.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:26.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:26.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:26.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:26.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:26.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:26.799 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:09:26.842 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:09:26.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:26.881 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:09:26.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:26.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:26.925 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:09:26.968 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:09:27.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:27.010 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:09:27.034 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:27.056 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:09:27.076 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 19 16 42529 42513 139.825 106.188 0.00361847 0.00714683 2024-03-04T16:09:27.100 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:09:27.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:09:27.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:27.169 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:27.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.296 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:27.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:27.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:27.433 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[29779]: pgmap v278: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[29779]: osdmap e174: 8 total, 8 up, 8 in 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3984814683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[38113]: pgmap v278: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-04T16:09:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:27.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:27.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[38113]: osdmap e174: 8 total, 8 up, 8 in 2024-03-04T16:09:27.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:27 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3984814683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:27.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:27.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:27 smithi176 ceph-mon[33619]: pgmap v278: 148 pgs: 148 active+clean; 2.1 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-04T16:09:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:27 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:27 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:27 smithi176 ceph-mon[33619]: osdmap e174: 8 total, 8 up, 8 in 2024-03-04T16:09:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:27 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3984814683' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:27.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:27.720 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:27.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:27.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:27.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:27.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:27.897 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:27.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:28.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:28.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:09:28.077479+0000 min lat: 0.00119552 max lat: 0.122782 avg lat: 0.00723629 2024-03-04T16:09:28.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:09:28.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 20 16 44213 44197 138.096 105.25 0.00813645 0.00723629 2024-03-04T16:09:28.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:09:28.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:28.180 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:28.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:28.282 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:28.314 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:28.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:28.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:28.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:28.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:28.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:28.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.729 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:28.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:28.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:28.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:28.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:28.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:29.002 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:29.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 21 16 45956 45940 136.707 108.938 0.00368331 0.00730958 2024-03-04T16:09:29.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.145 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:29.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:29.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.293 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:29.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:29.381 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:29 smithi038 ceph-mon[29779]: pgmap v280: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-04T16:09:29.381 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:29 smithi038 ceph-mon[38113]: pgmap v280: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-04T16:09:29.394 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:29.430 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:29.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.565 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:29.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:29 smithi176 ceph-mon[33619]: pgmap v280: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-04T16:09:29.632 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:29.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:29.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:29.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:29.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:29.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:29.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:30.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 22 15 47626 47611 135.24 104.438 0.00615398 0.00738894 2024-03-04T16:09:30.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:30.149 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:30.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:30.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:30.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:30.432 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:30.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:09:30.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.594 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:30.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:30.725 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:30.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:30.880 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:30.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:30.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:31.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:31.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 23 16 49243 49227 133.75 101 0.00697184 0.00747061 2024-03-04T16:09:31.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:31.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:31.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:31.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:31.328 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:31.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:31 smithi038 ceph-mon[29779]: pgmap v281: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-04T16:09:31.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:31 smithi038 ceph-mon[29779]: osdmap e175: 8 total, 8 up, 8 in 2024-03-04T16:09:31.377 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:31 smithi038 ceph-mon[38113]: pgmap v281: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-04T16:09:31.377 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:31 smithi038 ceph-mon[38113]: osdmap e175: 8 total, 8 up, 8 in 2024-03-04T16:09:31.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:31.466 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:31.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:31.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.604 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:31.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:31 smithi176 ceph-mon[33619]: pgmap v281: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-04T16:09:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:31 smithi176 ceph-mon[33619]: osdmap e175: 8 total, 8 up, 8 in 2024-03-04T16:09:31.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:31.743 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:31.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:31.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:31.883 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:31.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:31.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:32.021 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:32.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 24 16 50943 50927 132.604 106.25 0.00627081 0.00753617 2024-03-04T16:09:32.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:32.174 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:49694 8.0 8:0a7149bc:::benchmark_data_smithi038_107057_object49693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:09:31.366238+0000", 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0095588380000000001, 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095951539999999998, 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:09:32.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 49694 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366238+0000", 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.201 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366238+0000", 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366240+0000", 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3470000000000001e-06 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:09:32.202 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366264+0000", 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3986999999999999e-05 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366275+0000", 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1028e-05 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366283+0000", 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9230000000000002e-06 2024-03-04T16:09:32.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.368790+0000", 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025070159999999999 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:09:32.204 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.368913+0000", 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000122299 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,5", 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.369051+0000", 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013882799999999999 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375099+0000", 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060474789999999997 2024-03-04T16:09:32.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375132+0000", 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3351e-05 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:49696 8.0 8:0e1fe2c9:::benchmark_data_smithi038_107057_object49695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:09:31.366667+0000", 2024-03-04T16:09:32.206 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0091297319999999998, 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0092730499999999997, 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 49696 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.207 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366667+0000", 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366667+0000", 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:09:32.208 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366668+0000", 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4190000000000001e-06 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366747+0000", 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8881999999999995e-05 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366762+0000", 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4939e-05 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.209 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.366771+0000", 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6410000000000008e-06 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.372519+0000", 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057481759999999998 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:09:32.210 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.372669+0000", 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000149796 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,5", 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.372828+0000", 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015887599999999999 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375139+0000", 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023111160000000002 2024-03-04T16:09:32.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375437+0000", 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00029855300000000001 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15171.0:49707 8.7 e175/167 8:e253f6a7:::benchmark_data_smithi038_107057_object49706:head v 175'3126, mlcod=175'3123)", 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:09:31.375321+0000", 2024-03-04T16:09:32.212 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00047588200000000001, 2024-03-04T16:09:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00073606699999999997, 2024-03-04T16:09:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:09:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:09:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:09:32.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375321+0000", 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375321+0000", 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:09:32.214 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375322+0000", 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0800000000000004e-07 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375343+0000", 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1691999999999998e-05 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375351+0000", 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5820000000000004e-06 2024-03-04T16:09:32.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375357+0000", 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7370000000000003e-06 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375445+0000", 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8627000000000004e-05 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375466+0000", 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0664000000000001e-05 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:49710 8.b 8:d4fb7864:::benchmark_data_smithi038_107057_object49709:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:09:31.375769+0000", 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.8008999999999999e-05, 2024-03-04T16:09:32.217 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00037867100000000002, 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 49710 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375769+0000", 2024-03-04T16:09:32.218 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375769+0000", 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375771+0000", 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9360000000000002e-06 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.219 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375794+0000", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3453000000000001e-05 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375807+0000", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3234999999999999e-05 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375816+0000", 2024-03-04T16:09:32.220 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5210000000000002e-06 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.375881+0000", 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5581999999999997e-05 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.376020+0000", 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000138465 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:09:32.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:09:31.376154+0000", 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013367300000000001 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 4 2024-03-04T16:09:32.222 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:09:32.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:32.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:32.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.445 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:32.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:32.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:32.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:32.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:32.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:32.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:32.865 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:32.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:32.967 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:32.990 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:32.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:33.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 25 16 52445 52429 131.054 93.875 0.00483085 0.00762495 2024-03-04T16:09:33.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:33.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:33.120 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:33.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:33.254 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:33.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:33.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:33.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:33.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:33 smithi038 ceph-mon[29779]: pgmap v283: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 740 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-04T16:09:33.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:33 smithi038 ceph-mon[38113]: pgmap v283: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 740 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-04T16:09:33.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:33.524 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:33 smithi176 ceph-mon[33619]: pgmap v283: 148 pgs: 148 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 740 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-04T16:09:33.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:33.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:09:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:33.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:33.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.806 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:33.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:33.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:33.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:33.933 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:33.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:34.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:34.069 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:34.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:34.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 26 16 53862 53846 129.42 88.5625 0.00880245 0.00772112 2024-03-04T16:09:34.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:34.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:34.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:34.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:34.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:34.491 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:34.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:34.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:34.613 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:34.644 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:34.746 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:34.776 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:34.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:34.897 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:09:34.908 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:09:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:34.919 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:34.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:35.020 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.055 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:35.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:35.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 27 16 55299 55283 127.953 89.8125 0.0114792 0.00781051 2024-03-04T16:09:35.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:35.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:35.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:35.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:35.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:35.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[29779]: pgmap v284: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-04T16:09:35.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[29779]: from='client.25060 v1:172.21.15.38:0/1737494235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:35.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[29779]: osdmap e176: 8 total, 8 up, 8 in 2024-03-04T16:09:35.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[38113]: pgmap v284: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-04T16:09:35.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[38113]: from='client.25060 v1:172.21.15.38:0/1737494235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:35.453 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:35 smithi038 ceph-mon[38113]: osdmap e176: 8 total, 8 up, 8 in 2024-03-04T16:09:35.471 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:35.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:35.602 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:35 smithi176 ceph-mon[33619]: pgmap v284: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-03-04T16:09:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:35 smithi176 ceph-mon[33619]: from='client.25060 v1:172.21.15.38:0/1737494235' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:35 smithi176 ceph-mon[33619]: osdmap e176: 8 total, 8 up, 8 in 2024-03-04T16:09:35.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:35.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:35.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:35.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:35.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:35.978 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:36.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:36.077 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 28 16 56740 56724 126.599 90.0625 0.0146334 0.00789324 2024-03-04T16:09:36.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:36.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:36.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.284 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:36.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:36.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:36.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:36 smithi038 ceph-mon[29779]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.90k op/s 2024-03-04T16:09:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:36 smithi038 ceph-mon[38113]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.90k op/s 2024-03-04T16:09:36.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:36.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:36 smithi176 ceph-mon[33619]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.90k op/s 2024-03-04T16:09:36.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:36.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:36.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:36.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:36.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:36.948 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:36.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:37.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:37.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 29 15 58005 57990 124.962 79.125 0.00813642 0.00799749 2024-03-04T16:09:37.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:37.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:37.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:37.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:37.374 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:37.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: osdmap e177: 8 total, 8 up, 8 in 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: osdmap e177: 8 total, 8 up, 8 in 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:37.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:37.520 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: osdmap e177: 8 total, 8 up, 8 in 2024-03-04T16:09:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:37.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:09:37.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:37.650 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:37.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.777 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:37.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:37.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:37.912 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:37.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:37.924 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:09:38.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:38.051 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:38.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 30 16 59469 59453 123.844 91.4375 0.0046555 0.00806969 2024-03-04T16:09:38.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:38.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:38.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:38.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:38.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:38.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:38.467 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[29779]: pgmap v288: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[29779]: osdmap e178: 8 total, 8 up, 8 in 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[38113]: pgmap v288: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:38.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:38 smithi038 ceph-mon[38113]: osdmap e178: 8 total, 8 up, 8 in 2024-03-04T16:09:38.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:38 smithi176 ceph-mon[33619]: pgmap v288: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 706 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:38 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4195783577' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:09:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:38 smithi176 ceph-mon[33619]: osdmap e178: 8 total, 8 up, 8 in 2024-03-04T16:09:38.610 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:38.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.751 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:38.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:38.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:38.887 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:38.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:38.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:38.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:39.032 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:39.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 31 16 60957 60941 122.849 93 0.00548483 0.00813478 2024-03-04T16:09:39.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:39.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:39.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:39.300 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:39.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:39.436 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:39 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:39 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:39 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:39 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:39.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:39.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:39 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:39 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:39.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:39.709 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:39.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:39.847 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:39.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:39.981 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:39.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:40.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 32 16 62405 62389 121.838 90.5 0.0103381 0.00820235 2024-03-04T16:09:40.083 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:40.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:40.218 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":284,"stamp":"2024-03-04T16:09:33.424304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3402635614,"num_objects":52109,"num_object_clones":0,"num_object_copies":156327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52109,"num_whiteouts":0,"num_read":1325,"num_read_kb":1171,"num_write":52526,"num_write_kb":3324765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2956616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52569,"ondisk_log_size":52569,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11279592,"kb_used_data":9833500,"kb_used_omap":12,"kb_used_meta":1446003,"kb_avail":738681624,"statfs":{"total":767960285184,"available":756409982976,"internally_reserved":0,"allocated":10069504000,"data_stored":10066019754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1480707634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,10,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1142161408,"num_objects":17428,"num_object_clones":0,"num_object_copies":52284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17428,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":17428,"num_write_kb":1115392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003689"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.175914+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:09:30.175914+0000","last_peered":"2024-03-04T16:09:30.175914+0000","last_clean":"2024-03-04T16:09:30.175914+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:09:30.175914+0000","last_undegraded":"2024-03-04T16:09:30.175914+0000","last_fullsized":"2024-03-04T16:09:30.175914+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:44.024543+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186831+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:09:30.186831+0000","last_peered":"2024-03-04T16:09:30.186831+0000","last_clean":"2024-03-04T16:09:30.186831+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:09:30.186831+0000","last_undegraded":"2024-03-04T16:09:30.186831+0000","last_fullsized":"2024-03-04T16:09:30.186831+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.583278+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183908+0000","last_change":"2024-03-04T16:08:56.062397+0000","last_active":"2024-03-04T16:09:30.183908+0000","last_peered":"2024-03-04T16:09:30.183908+0000","last_clean":"2024-03-04T16:09:30.183908+0000","last_became_active":"2024-03-04T16:07:36.205930+0000","last_became_peered":"2024-03-04T16:07:36.205930+0000","last_unstale":"2024-03-04T16:09:30.183908+0000","last_undegraded":"2024-03-04T16:09:30.183908+0000","last_fullsized":"2024-03-04T16:09:30.183908+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.471716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":263,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.175728+0000","last_change":"2024-03-04T16:08:15.752830+0000","last_active":"2024-03-04T16:09:30.175728+0000","last_peered":"2024-03-04T16:09:30.175728+0000","last_clean":"2024-03-04T16:09:30.175728+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:09:30.175728+0000","last_undegraded":"2024-03-04T16:09:30.175728+0000","last_fullsized":"2024-03-04T16:09:30.175728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:15.752737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.752737+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.187986+0000","last_change":"2024-03-04T16:09:20.902067+0000","last_active":"2024-03-04T16:09:30.187986+0000","last_peered":"2024-03-04T16:09:30.187986+0000","last_clean":"2024-03-04T16:09:30.187986+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:09:30.187986+0000","last_undegraded":"2024-03-04T16:09:30.187986+0000","last_fullsized":"2024-03-04T16:09:30.187986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:34.490944+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.175855+0000","last_change":"2024-03-04T16:09:00.760314+0000","last_active":"2024-03-04T16:09:30.175855+0000","last_peered":"2024-03-04T16:09:30.175855+0000","last_clean":"2024-03-04T16:09:30.175855+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:09:30.175855+0000","last_undegraded":"2024-03-04T16:09:30.175855+0000","last_fullsized":"2024-03-04T16:09:30.175855+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.190987+0000","last_change":"2024-03-04T16:08:30.752343+0000","last_active":"2024-03-04T16:09:30.190987+0000","last_peered":"2024-03-04T16:09:30.190987+0000","last_clean":"2024-03-04T16:09:30.190987+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:09:30.190987+0000","last_undegraded":"2024-03-04T16:09:30.190987+0000","last_fullsized":"2024-03-04T16:09:30.190987+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.745969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":316,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186435+0000","last_change":"2024-03-04T16:08:21.634801+0000","last_active":"2024-03-04T16:09:30.186435+0000","last_peered":"2024-03-04T16:09:30.186435+0000","last_clean":"2024-03-04T16:09:30.186435+0000","last_became_active":"2024-03-04T16:07:36.221904+0000","last_became_peered":"2024-03-04T16:07:36.221904+0000","last_unstale":"2024-03-04T16:09:30.186435+0000","last_undegraded":"2024-03-04T16:09:30.186435+0000","last_fullsized":"2024-03-04T16:09:30.186435+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.436614+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179688+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:09:30.179688+0000","last_peered":"2024-03-04T16:09:30.179688+0000","last_clean":"2024-03-04T16:09:30.179688+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:09:30.179688+0000","last_undegraded":"2024-03-04T16:09:30.179688+0000","last_fullsized":"2024-03-04T16:09:30.179688+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:35.559365+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":242,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.180330+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:09:30.180330+0000","last_peered":"2024-03-04T16:09:30.180330+0000","last_clean":"2024-03-04T16:09:30.180330+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:09:30.180330+0000","last_undegraded":"2024-03-04T16:09:30.180330+0000","last_fullsized":"2024-03-04T16:09:30.180330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.564603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":258,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177950+0000","last_change":"2024-03-04T16:08:18.150034+0000","last_active":"2024-03-04T16:09:30.177950+0000","last_peered":"2024-03-04T16:09:30.177950+0000","last_clean":"2024-03-04T16:09:30.177950+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:09:30.177950+0000","last_undegraded":"2024-03-04T16:09:30.177950+0000","last_fullsized":"2024-03-04T16:09:30.177950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.921681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":299,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176245+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:09:30.176245+0000","last_peered":"2024-03-04T16:09:30.176245+0000","last_clean":"2024-03-04T16:09:30.176245+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:09:30.176245+0000","last_undegraded":"2024-03-04T16:09:30.176245+0000","last_fullsized":"2024-03-04T16:09:30.176245+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":246,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177198+0000","last_change":"2024-03-04T16:08:27.210748+0000","last_active":"2024-03-04T16:09:30.177198+0000","last_peered":"2024-03-04T16:09:30.177198+0000","last_clean":"2024-03-04T16:09:30.177198+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:09:30.177198+0000","last_undegraded":"2024-03-04T16:09:30.177198+0000","last_fullsized":"2024-03-04T16:09:30.177198+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.210666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.210666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181955+0000","last_change":"2024-03-04T16:08:22.797116+0000","last_active":"2024-03-04T16:09:30.181955+0000","last_peered":"2024-03-04T16:09:30.181955+0000","last_clean":"2024-03-04T16:09:30.181955+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:09:30.181955+0000","last_undegraded":"2024-03-04T16:09:30.181955+0000","last_fullsized":"2024-03-04T16:09:30.181955+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.796991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.796991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.136282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":250,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179774+0000","last_change":"2024-03-04T16:08:07.153510+0000","last_active":"2024-03-04T16:09:30.179774+0000","last_peered":"2024-03-04T16:09:30.179774+0000","last_clean":"2024-03-04T16:09:30.179774+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:09:30.179774+0000","last_undegraded":"2024-03-04T16:09:30.179774+0000","last_fullsized":"2024-03-04T16:09:30.179774+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:07.153434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.153434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.234376+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":271,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176733+0000","last_change":"2024-03-04T16:09:04.791187+0000","last_active":"2024-03-04T16:09:30.176733+0000","last_peered":"2024-03-04T16:09:30.176733+0000","last_clean":"2024-03-04T16:09:30.176733+0000","last_became_active":"2024-03-04T16:04:22.732082+0000","last_became_peered":"2024-03-04T16:04:22.732082+0000","last_unstale":"2024-03-04T16:09:30.176733+0000","last_undegraded":"2024-03-04T16:09:30.176733+0000","last_fullsized":"2024-03-04T16:09:30.176733+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:14.516520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":240,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176354+0000","last_change":"2024-03-04T16:09:03.790808+0000","last_active":"2024-03-04T16:09:30.176354+0000","last_peered":"2024-03-04T16:09:30.176354+0000","last_clean":"2024-03-04T16:09:30.176354+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:09:30.176354+0000","last_undegraded":"2024-03-04T16:09:30.176354+0000","last_fullsized":"2024-03-04T16:09:30.176354+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:13.192635+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.184686+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:09:30.184686+0000","last_peered":"2024-03-04T16:09:30.184686+0000","last_clean":"2024-03-04T16:09:30.184686+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:09:30.184686+0000","last_undegraded":"2024-03-04T16:09:30.184686+0000","last_fullsized":"2024-03-04T16:09:30.184686+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.853952+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179140+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:09:30.179140+0000","last_peered":"2024-03-04T16:09:30.179140+0000","last_clean":"2024-03-04T16:09:30.179140+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:09:30.179140+0000","last_undegraded":"2024-03-04T16:09:30.179140+0000","last_fullsized":"2024-03-04T16:09:30.179140+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.718715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":280,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182604+0000","last_change":"2024-03-04T16:08:27.760340+0000","last_active":"2024-03-04T16:09:30.182604+0000","last_peered":"2024-03-04T16:09:30.182604+0000","last_clean":"2024-03-04T16:09:30.182604+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:09:30.182604+0000","last_undegraded":"2024-03-04T16:09:30.182604+0000","last_fullsized":"2024-03-04T16:09:30.182604+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:27.760194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.760194+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.308820+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":232,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.193698+0000","last_change":"2024-03-04T16:06:26.278970+0000","last_active":"2024-03-04T16:09:30.193698+0000","last_peered":"2024-03-04T16:09:30.193698+0000","last_clean":"2024-03-04T16:09:30.193698+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:09:30.193698+0000","last_undegraded":"2024-03-04T16:09:30.193698+0000","last_fullsized":"2024-03-04T16:09:30.193698+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:26.278824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:26.278824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176898+0000","last_change":"2024-03-04T16:08:04.014351+0000","last_active":"2024-03-04T16:09:30.176898+0000","last_peered":"2024-03-04T16:09:30.176898+0000","last_clean":"2024-03-04T16:09:30.176898+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:09:30.176898+0000","last_undegraded":"2024-03-04T16:09:30.176898+0000","last_fullsized":"2024-03-04T16:09:30.176898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:04.014236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.014236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":259,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176353+0000","last_change":"2024-03-04T16:09:23.836557+0000","last_active":"2024-03-04T16:09:30.176353+0000","last_peered":"2024-03-04T16:09:30.176353+0000","last_clean":"2024-03-04T16:09:30.176353+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:09:30.176353+0000","last_undegraded":"2024-03-04T16:09:30.176353+0000","last_fullsized":"2024-03-04T16:09:30.176353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:43.944821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":270,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.191106+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:09:30.191106+0000","last_peered":"2024-03-04T16:09:30.191106+0000","last_clean":"2024-03-04T16:09:30.191106+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:09:30.191106+0000","last_undegraded":"2024-03-04T16:09:30.191106+0000","last_fullsized":"2024-03-04T16:09:30.191106+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.113960+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":241,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182583+0000","last_change":"2024-03-04T16:09:01.793018+0000","last_active":"2024-03-04T16:09:30.182583+0000","last_peered":"2024-03-04T16:09:30.182583+0000","last_clean":"2024-03-04T16:09:30.182583+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:09:30.182583+0000","last_undegraded":"2024-03-04T16:09:30.182583+0000","last_fullsized":"2024-03-04T16:09:30.182583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:09:01.792870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.792870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:29.041563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"124'9","reported_seq":346,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182338+0000","last_change":"2024-03-04T16:08:20.850258+0000","last_active":"2024-03-04T16:09:30.182338+0000","last_peered":"2024-03-04T16:09:30.182338+0000","last_clean":"2024-03-04T16:09:30.182338+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:09:30.182338+0000","last_undegraded":"2024-03-04T16:09:30.182338+0000","last_fullsized":"2024-03-04T16:09:30.182338+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:08:20.850137+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:06:42.058461+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.850137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.862934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186911+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:09:30.186911+0000","last_peered":"2024-03-04T16:09:30.186911+0000","last_clean":"2024-03-04T16:09:30.186911+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:09:30.186911+0000","last_undegraded":"2024-03-04T16:09:30.186911+0000","last_fullsized":"2024-03-04T16:09:30.186911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.583243+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":99,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.189391+0000","last_change":"2024-03-04T16:08:04.989047+0000","last_active":"2024-03-04T16:09:30.189391+0000","last_peered":"2024-03-04T16:09:30.189391+0000","last_clean":"2024-03-04T16:09:30.189391+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:09:30.189391+0000","last_undegraded":"2024-03-04T16:09:30.189391+0000","last_fullsized":"2024-03-04T16:09:30.189391+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:08:04.988877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.988877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.184850+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:09:30.184850+0000","last_peered":"2024-03-04T16:09:30.184850+0000","last_clean":"2024-03-04T16:09:30.184850+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:09:30.184850+0000","last_undegraded":"2024-03-04T16:09:30.184850+0000","last_fullsized":"2024-03-04T16:09:30.184850+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.336787+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.189621+0000","last_change":"2024-03-04T16:09:01.127663+0000","last_active":"2024-03-04T16:09:30.189621+0000","last_peered":"2024-03-04T16:09:30.189621+0000","last_clean":"2024-03-04T16:09:30.189621+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:09:30.189621+0000","last_undegraded":"2024-03-04T16:09:30.189621+0000","last_fullsized":"2024-03-04T16:09:30.189621+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:01.883191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":263,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186190+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:09:30.186190+0000","last_peered":"2024-03-04T16:09:30.186190+0000","last_clean":"2024-03-04T16:09:30.186190+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:09:30.186190+0000","last_undegraded":"2024-03-04T16:09:30.186190+0000","last_fullsized":"2024-03-04T16:09:30.186190+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.074160+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":293,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182387+0000","last_change":"2024-03-04T16:08:58.094226+0000","last_active":"2024-03-04T16:09:30.182387+0000","last_peered":"2024-03-04T16:09:30.182387+0000","last_clean":"2024-03-04T16:09:30.182387+0000","last_became_active":"2024-03-04T16:04:22.734590+0000","last_became_peered":"2024-03-04T16:04:22.734590+0000","last_unstale":"2024-03-04T16:09:30.182387+0000","last_undegraded":"2024-03-04T16:09:30.182387+0000","last_fullsized":"2024-03-04T16:09:30.182387+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.639004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181859+0000","last_change":"2024-03-04T16:09:19.779562+0000","last_active":"2024-03-04T16:09:30.181859+0000","last_peered":"2024-03-04T16:09:30.181859+0000","last_clean":"2024-03-04T16:09:30.181859+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:09:30.181859+0000","last_undegraded":"2024-03-04T16:09:30.181859+0000","last_fullsized":"2024-03-04T16:09:30.181859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:09:19.779373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:27.504868+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181996+0000","last_change":"2024-03-04T16:08:56.035409+0000","last_active":"2024-03-04T16:09:30.181996+0000","last_peered":"2024-03-04T16:09:30.181996+0000","last_clean":"2024-03-04T16:09:30.181996+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:09:30.181996+0000","last_undegraded":"2024-03-04T16:09:30.181996+0000","last_fullsized":"2024-03-04T16:09:30.181996+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.644040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183812+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:09:30.183812+0000","last_peered":"2024-03-04T16:09:30.183812+0000","last_clean":"2024-03-04T16:09:30.183812+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:09:30.183812+0000","last_undegraded":"2024-03-04T16:09:30.183812+0000","last_fullsized":"2024-03-04T16:09:30.183812+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.356777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":245,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186300+0000","last_change":"2024-03-04T16:06:24.244107+0000","last_active":"2024-03-04T16:09:30.186300+0000","last_peered":"2024-03-04T16:09:30.186300+0000","last_clean":"2024-03-04T16:09:30.186300+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:09:30.186300+0000","last_undegraded":"2024-03-04T16:09:30.186300+0000","last_fullsized":"2024-03-04T16:09:30.186300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:06:24.243967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:24.243967+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.188165+0000","last_change":"2024-03-04T16:08:06.354795+0000","last_active":"2024-03-04T16:09:30.188165+0000","last_peered":"2024-03-04T16:09:30.188165+0000","last_clean":"2024-03-04T16:09:30.188165+0000","last_became_active":"2024-03-04T16:07:36.224625+0000","last_became_peered":"2024-03-04T16:07:36.224625+0000","last_unstale":"2024-03-04T16:09:30.188165+0000","last_undegraded":"2024-03-04T16:09:30.188165+0000","last_fullsized":"2024-03-04T16:09:30.188165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181951+0000","last_change":"2024-03-04T16:08:58.788323+0000","last_active":"2024-03-04T16:09:30.181951+0000","last_peered":"2024-03-04T16:09:30.181951+0000","last_clean":"2024-03-04T16:09:30.181951+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:09:30.181951+0000","last_undegraded":"2024-03-04T16:09:30.181951+0000","last_fullsized":"2024-03-04T16:09:30.181951+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.788235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.788235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":258,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177671+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:09:30.177671+0000","last_peered":"2024-03-04T16:09:30.177671+0000","last_clean":"2024-03-04T16:09:30.177671+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:09:30.177671+0000","last_undegraded":"2024-03-04T16:09:30.177671+0000","last_fullsized":"2024-03-04T16:09:30.177671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.344720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":258,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186329+0000","last_change":"2024-03-04T16:06:34.283125+0000","last_active":"2024-03-04T16:09:30.186329+0000","last_peered":"2024-03-04T16:09:30.186329+0000","last_clean":"2024-03-04T16:09:30.186329+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:09:30.186329+0000","last_undegraded":"2024-03-04T16:09:30.186329+0000","last_fullsized":"2024-03-04T16:09:30.186329+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:06:34.282977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:34.282977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179787+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:09:30.179787+0000","last_peered":"2024-03-04T16:09:30.179787+0000","last_clean":"2024-03-04T16:09:30.179787+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:09:30.179787+0000","last_undegraded":"2024-03-04T16:09:30.179787+0000","last_fullsized":"2024-03-04T16:09:30.179787+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.303288+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.187887+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:09:30.187887+0000","last_peered":"2024-03-04T16:09:30.187887+0000","last_clean":"2024-03-04T16:09:30.187887+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:09:30.187887+0000","last_undegraded":"2024-03-04T16:09:30.187887+0000","last_fullsized":"2024-03-04T16:09:30.187887+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:57.871591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183057+0000","last_change":"2024-03-04T16:09:13.858427+0000","last_active":"2024-03-04T16:09:30.183057+0000","last_peered":"2024-03-04T16:09:30.183057+0000","last_clean":"2024-03-04T16:09:30.183057+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:09:30.183057+0000","last_undegraded":"2024-03-04T16:09:30.183057+0000","last_fullsized":"2024-03-04T16:09:30.183057+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.858316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.858316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:16.795714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":275,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185553+0000","last_change":"2024-03-04T16:09:13.635317+0000","last_active":"2024-03-04T16:09:30.185553+0000","last_peered":"2024-03-04T16:09:30.185553+0000","last_clean":"2024-03-04T16:09:30.185553+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:09:30.185553+0000","last_undegraded":"2024-03-04T16:09:30.185553+0000","last_fullsized":"2024-03-04T16:09:30.185553+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:31.274560+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":243,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185976+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:09:30.185976+0000","last_peered":"2024-03-04T16:09:30.185976+0000","last_clean":"2024-03-04T16:09:30.185976+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:09:30.185976+0000","last_undegraded":"2024-03-04T16:09:30.185976+0000","last_fullsized":"2024-03-04T16:09:30.185976+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.513501+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185348+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:09:30.185348+0000","last_peered":"2024-03-04T16:09:30.185348+0000","last_clean":"2024-03-04T16:09:30.185348+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:09:30.185348+0000","last_undegraded":"2024-03-04T16:09:30.185348+0000","last_fullsized":"2024-03-04T16:09:30.185348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:28.419644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176339+0000","last_change":"2024-03-04T16:07:31.213593+0000","last_active":"2024-03-04T16:09:30.176339+0000","last_peered":"2024-03-04T16:09:30.176339+0000","last_clean":"2024-03-04T16:09:30.176339+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:09:30.176339+0000","last_undegraded":"2024-03-04T16:09:30.176339+0000","last_fullsized":"2024-03-04T16:09:30.176339+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:31.213444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:07:31.213444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":274,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176224+0000","last_change":"2024-03-04T16:09:02.785333+0000","last_active":"2024-03-04T16:09:30.176224+0000","last_peered":"2024-03-04T16:09:30.176224+0000","last_clean":"2024-03-04T16:09:30.176224+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:09:30.176224+0000","last_undegraded":"2024-03-04T16:09:30.176224+0000","last_fullsized":"2024-03-04T16:09:30.176224+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.439534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":121,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182899+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:09:30.182899+0000","last_peered":"2024-03-04T16:09:30.182899+0000","last_clean":"2024-03-04T16:09:30.182899+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:09:30.182899+0000","last_undegraded":"2024-03-04T16:09:30.182899+0000","last_fullsized":"2024-03-04T16:09:30.182899+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:41.311717+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":354,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.178043+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:09:30.178043+0000","last_peered":"2024-03-04T16:09:30.178043+0000","last_clean":"2024-03-04T16:09:30.178043+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:09:30.178043+0000","last_undegraded":"2024-03-04T16:09:30.178043+0000","last_fullsized":"2024-03-04T16:09:30.178043+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:37.704629+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":265,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182236+0000","last_change":"2024-03-04T16:09:18.820910+0000","last_active":"2024-03-04T16:09:30.182236+0000","last_peered":"2024-03-04T16:09:30.182236+0000","last_clean":"2024-03-04T16:09:30.182236+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:09:30.182236+0000","last_undegraded":"2024-03-04T16:09:30.182236+0000","last_fullsized":"2024-03-04T16:09:30.182236+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:18.820782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.820782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:44.116052+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":272,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186145+0000","last_change":"2024-03-04T16:09:17.965043+0000","last_active":"2024-03-04T16:09:30.186145+0000","last_peered":"2024-03-04T16:09:30.186145+0000","last_clean":"2024-03-04T16:09:30.186145+0000","last_became_active":"2024-03-04T16:04:22.731236+0000","last_became_peered":"2024-03-04T16:04:22.731236+0000","last_unstale":"2024-03-04T16:09:30.186145+0000","last_undegraded":"2024-03-04T16:09:30.186145+0000","last_fullsized":"2024-03-04T16:09:30.186145+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:47.953992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176513+0000","last_change":"2024-03-04T16:08:27.876805+0000","last_active":"2024-03-04T16:09:30.176513+0000","last_peered":"2024-03-04T16:09:30.176513+0000","last_clean":"2024-03-04T16:09:30.176513+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:09:30.176513+0000","last_undegraded":"2024-03-04T16:09:30.176513+0000","last_fullsized":"2024-03-04T16:09:30.176513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.279216+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":326,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.190812+0000","last_change":"2024-03-04T16:08:07.413971+0000","last_active":"2024-03-04T16:09:30.190812+0000","last_peered":"2024-03-04T16:09:30.190812+0000","last_clean":"2024-03-04T16:09:30.190812+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:09:30.190812+0000","last_undegraded":"2024-03-04T16:09:30.190812+0000","last_fullsized":"2024-03-04T16:09:30.190812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:34.743520+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":145,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182043+0000","last_change":"2024-03-04T16:08:57.778070+0000","last_active":"2024-03-04T16:09:30.182043+0000","last_peered":"2024-03-04T16:09:30.182043+0000","last_clean":"2024-03-04T16:09:30.182043+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:09:30.182043+0000","last_undegraded":"2024-03-04T16:09:30.182043+0000","last_fullsized":"2024-03-04T16:09:30.182043+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:57.777958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.777958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.367940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":270,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.178218+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:09:30.178218+0000","last_peered":"2024-03-04T16:09:30.178218+0000","last_clean":"2024-03-04T16:09:30.178218+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:09:30.178218+0000","last_undegraded":"2024-03-04T16:09:30.178218+0000","last_fullsized":"2024-03-04T16:09:30.178218+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.493416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182902+0000","last_change":"2024-03-04T16:08:12.603101+0000","last_active":"2024-03-04T16:09:30.182902+0000","last_peered":"2024-03-04T16:09:30.182902+0000","last_clean":"2024-03-04T16:09:30.182902+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:09:30.182902+0000","last_undegraded":"2024-03-04T16:09:30.182902+0000","last_fullsized":"2024-03-04T16:09:30.182902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.602969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.602969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:38.139716+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":235,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.193610+0000","last_change":"2024-03-04T16:06:36.235353+0000","last_active":"2024-03-04T16:09:30.193610+0000","last_peered":"2024-03-04T16:09:30.193610+0000","last_clean":"2024-03-04T16:09:30.193610+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:09:30.193610+0000","last_undegraded":"2024-03-04T16:09:30.193610+0000","last_fullsized":"2024-03-04T16:09:30.193610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:36.235220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:36.235220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181534+0000","last_change":"2024-03-04T16:09:07.827652+0000","last_active":"2024-03-04T16:09:30.181534+0000","last_peered":"2024-03-04T16:09:30.181534+0000","last_clean":"2024-03-04T16:09:30.181534+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:09:30.181534+0000","last_undegraded":"2024-03-04T16:09:30.181534+0000","last_fullsized":"2024-03-04T16:09:30.181534+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:11.354299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":97,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.189478+0000","last_change":"2024-03-04T16:08:20.028471+0000","last_active":"2024-03-04T16:09:30.189478+0000","last_peered":"2024-03-04T16:09:30.189478+0000","last_clean":"2024-03-04T16:09:30.189478+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:09:30.189478+0000","last_undegraded":"2024-03-04T16:09:30.189478+0000","last_fullsized":"2024-03-04T16:09:30.189478+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:20.025566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.025566+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.552865+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182929+0000","last_change":"2024-03-04T16:08:19.818086+0000","last_active":"2024-03-04T16:09:30.182929+0000","last_peered":"2024-03-04T16:09:30.182929+0000","last_clean":"2024-03-04T16:09:30.182929+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:09:30.182929+0000","last_undegraded":"2024-03-04T16:09:30.182929+0000","last_fullsized":"2024-03-04T16:09:30.182929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.818005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.818005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:35.179560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":250,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177581+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:09:30.177581+0000","last_peered":"2024-03-04T16:09:30.177581+0000","last_clean":"2024-03-04T16:09:30.177581+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:09:30.177581+0000","last_undegraded":"2024-03-04T16:09:30.177581+0000","last_fullsized":"2024-03-04T16:09:30.177581+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:19.621914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":271,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.180915+0000","last_change":"2024-03-04T16:08:18.893496+0000","last_active":"2024-03-04T16:09:30.180915+0000","last_peered":"2024-03-04T16:09:30.180915+0000","last_clean":"2024-03-04T16:09:30.180915+0000","last_became_active":"2024-03-04T16:07:36.206947+0000","last_became_peered":"2024-03-04T16:07:36.206947+0000","last_unstale":"2024-03-04T16:09:30.180915+0000","last_undegraded":"2024-03-04T16:09:30.180915+0000","last_fullsized":"2024-03-04T16:09:30.180915+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.893370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.893370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.386307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.193786+0000","last_change":"2024-03-04T16:06:23.314048+0000","last_active":"2024-03-04T16:09:30.193786+0000","last_peered":"2024-03-04T16:09:30.193786+0000","last_clean":"2024-03-04T16:09:30.193786+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:09:30.193786+0000","last_undegraded":"2024-03-04T16:09:30.193786+0000","last_fullsized":"2024-03-04T16:09:30.193786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:06:23.313799+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:06:23.313799+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"175'3218","reported_seq":3244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.367120+0000","last_change":"2024-03-04T16:08:56.231065+0000","last_active":"2024-03-04T16:09:32.367120+0000","last_peered":"2024-03-04T16:09:32.367120+0000","last_clean":"2024-03-04T16:09:32.367120+0000","last_became_active":"2024-03-04T16:08:56.230573+0000","last_became_peered":"2024-03-04T16:08:56.230573+0000","last_unstale":"2024-03-04T16:09:32.367120+0000","last_undegraded":"2024-03-04T16:09:32.367120+0000","last_fullsized":"2024-03-04T16:09:32.367120+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3218,"log_dups_size":0,"ondisk_log_size":3218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.913860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210894848,"num_objects":3218,"num_object_clones":0,"num_object_copies":9654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3218,"num_write_kb":205952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177466+0000","last_change":"2024-03-04T16:08:56.044901+0000","last_active":"2024-03-04T16:09:30.177466+0000","last_peered":"2024-03-04T16:09:30.177466+0000","last_clean":"2024-03-04T16:09:30.177466+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:09:30.177466+0000","last_undegraded":"2024-03-04T16:09:30.177466+0000","last_fullsized":"2024-03-04T16:09:30.177466+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.356711+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":288,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182686+0000","last_change":"2024-03-04T16:08:32.803550+0000","last_active":"2024-03-04T16:09:30.182686+0000","last_peered":"2024-03-04T16:09:30.182686+0000","last_clean":"2024-03-04T16:09:30.182686+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:09:30.182686+0000","last_undegraded":"2024-03-04T16:09:30.182686+0000","last_fullsized":"2024-03-04T16:09:30.182686+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:08:32.803433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:32.803433+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.264477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":280,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177517+0000","last_change":"2024-03-04T16:08:26.222236+0000","last_active":"2024-03-04T16:09:30.177517+0000","last_peered":"2024-03-04T16:09:30.177517+0000","last_clean":"2024-03-04T16:09:30.177517+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:09:30.177517+0000","last_undegraded":"2024-03-04T16:09:30.177517+0000","last_fullsized":"2024-03-04T16:09:30.177517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.781685+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":334,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181753+0000","last_change":"2024-03-04T16:09:06.869919+0000","last_active":"2024-03-04T16:09:30.181753+0000","last_peered":"2024-03-04T16:09:30.181753+0000","last_clean":"2024-03-04T16:09:30.181753+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:09:30.181753+0000","last_undegraded":"2024-03-04T16:09:30.181753+0000","last_fullsized":"2024-03-04T16:09:30.181753+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:06.869771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.869771+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.331005+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"175'3317","reported_seq":3343,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.422630+0000","last_change":"2024-03-04T16:08:56.789776+0000","last_active":"2024-03-04T16:09:32.422630+0000","last_peered":"2024-03-04T16:09:32.422630+0000","last_clean":"2024-03-04T16:09:32.422630+0000","last_became_active":"2024-03-04T16:08:56.789279+0000","last_became_peered":"2024-03-04T16:08:56.789279+0000","last_unstale":"2024-03-04T16:09:32.422630+0000","last_undegraded":"2024-03-04T16:09:32.422630+0000","last_fullsized":"2024-03-04T16:09:32.422630+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3317,"log_dups_size":0,"ondisk_log_size":3317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.712859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217382912,"num_objects":3317,"num_object_clones":0,"num_object_copies":9951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3317,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3317,"num_write_kb":212288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":232,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.188117+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:09:30.188117+0000","last_peered":"2024-03-04T16:09:30.188117+0000","last_clean":"2024-03-04T16:09:30.188117+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:09:30.188117+0000","last_undegraded":"2024-03-04T16:09:30.188117+0000","last_fullsized":"2024-03-04T16:09:30.188117+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:35.751119+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183089+0000","last_change":"2024-03-04T16:08:29.793308+0000","last_active":"2024-03-04T16:09:30.183089+0000","last_peered":"2024-03-04T16:09:30.183089+0000","last_clean":"2024-03-04T16:09:30.183089+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:09:30.183089+0000","last_undegraded":"2024-03-04T16:09:30.183089+0000","last_fullsized":"2024-03-04T16:09:30.183089+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:29.793217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.793217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.590776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":271,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176121+0000","last_change":"2024-03-04T16:07:24.247724+0000","last_active":"2024-03-04T16:09:30.176121+0000","last_peered":"2024-03-04T16:09:30.176121+0000","last_clean":"2024-03-04T16:09:30.176121+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:09:30.176121+0000","last_undegraded":"2024-03-04T16:09:30.176121+0000","last_fullsized":"2024-03-04T16:09:30.176121+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:07:24.247330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:24.247330+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":264,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181612+0000","last_change":"2024-03-04T16:09:18.939448+0000","last_active":"2024-03-04T16:09:30.181612+0000","last_peered":"2024-03-04T16:09:30.181612+0000","last_clean":"2024-03-04T16:09:30.181612+0000","last_became_active":"2024-03-04T16:07:36.213031+0000","last_became_peered":"2024-03-04T16:07:36.213031+0000","last_unstale":"2024-03-04T16:09:30.181612+0000","last_undegraded":"2024-03-04T16:09:30.181612+0000","last_fullsized":"2024-03-04T16:09:30.181612+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:18.939310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.939310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:30.070633+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"175'3212","reported_seq":3238,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.946464+0000","last_change":"2024-03-04T16:08:56.789841+0000","last_active":"2024-03-04T16:09:32.946464+0000","last_peered":"2024-03-04T16:09:32.946464+0000","last_clean":"2024-03-04T16:09:32.946464+0000","last_became_active":"2024-03-04T16:08:56.789562+0000","last_became_peered":"2024-03-04T16:08:56.789562+0000","last_unstale":"2024-03-04T16:09:32.946464+0000","last_undegraded":"2024-03-04T16:09:32.946464+0000","last_fullsized":"2024-03-04T16:09:32.946464+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3212,"log_dups_size":0,"ondisk_log_size":3212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.078343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210501632,"num_objects":3212,"num_object_clones":0,"num_object_copies":9636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3212,"num_write_kb":205568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":240,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181566+0000","last_change":"2024-03-04T16:09:10.830111+0000","last_active":"2024-03-04T16:09:30.181566+0000","last_peered":"2024-03-04T16:09:30.181566+0000","last_clean":"2024-03-04T16:09:30.181566+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:09:30.181566+0000","last_undegraded":"2024-03-04T16:09:30.181566+0000","last_fullsized":"2024-03-04T16:09:30.181566+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.145188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181888+0000","last_change":"2024-03-04T16:08:22.679372+0000","last_active":"2024-03-04T16:09:30.181888+0000","last_peered":"2024-03-04T16:09:30.181888+0000","last_clean":"2024-03-04T16:09:30.181888+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:09:30.181888+0000","last_undegraded":"2024-03-04T16:09:30.181888+0000","last_fullsized":"2024-03-04T16:09:30.181888+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.679241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.679241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:49.797013+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183215+0000","last_change":"2024-03-04T16:08:23.747781+0000","last_active":"2024-03-04T16:09:30.183215+0000","last_peered":"2024-03-04T16:09:30.183215+0000","last_clean":"2024-03-04T16:09:30.183215+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:09:30.183215+0000","last_undegraded":"2024-03-04T16:09:30.183215+0000","last_fullsized":"2024-03-04T16:09:30.183215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.747649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.747649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:32.644950+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":295,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181479+0000","last_change":"2024-03-04T16:09:23.983219+0000","last_active":"2024-03-04T16:09:30.181479+0000","last_peered":"2024-03-04T16:09:30.181479+0000","last_clean":"2024-03-04T16:09:30.181479+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:09:30.181479+0000","last_undegraded":"2024-03-04T16:09:30.181479+0000","last_fullsized":"2024-03-04T16:09:30.181479+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.480220+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"175'3272","reported_seq":3298,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.729883+0000","last_change":"2024-03-04T16:08:56.234782+0000","last_active":"2024-03-04T16:09:32.729883+0000","last_peered":"2024-03-04T16:09:32.729883+0000","last_clean":"2024-03-04T16:09:32.729883+0000","last_became_active":"2024-03-04T16:08:56.234486+0000","last_became_peered":"2024-03-04T16:08:56.234486+0000","last_unstale":"2024-03-04T16:09:32.729883+0000","last_undegraded":"2024-03-04T16:09:32.729883+0000","last_fullsized":"2024-03-04T16:09:32.729883+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3272,"log_dups_size":0,"ondisk_log_size":3272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.872524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214433792,"num_objects":3272,"num_object_clones":0,"num_object_copies":9816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3272,"num_write_kb":209408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":121,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181340+0000","last_change":"2024-03-04T16:09:15.838949+0000","last_active":"2024-03-04T16:09:30.181340+0000","last_peered":"2024-03-04T16:09:30.181340+0000","last_clean":"2024-03-04T16:09:30.181340+0000","last_became_active":"2024-03-04T16:07:36.211991+0000","last_became_peered":"2024-03-04T16:07:36.211991+0000","last_unstale":"2024-03-04T16:09:30.181340+0000","last_undegraded":"2024-03-04T16:09:30.181340+0000","last_fullsized":"2024-03-04T16:09:30.181340+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.838866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.838866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:40.043588+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182095+0000","last_change":"2024-03-04T16:08:20.664979+0000","last_active":"2024-03-04T16:09:30.182095+0000","last_peered":"2024-03-04T16:09:30.182095+0000","last_clean":"2024-03-04T16:09:30.182095+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:09:30.182095+0000","last_undegraded":"2024-03-04T16:09:30.182095+0000","last_fullsized":"2024-03-04T16:09:30.182095+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.664848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:30.403776+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":259,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179819+0000","last_change":"2024-03-04T16:09:04.159447+0000","last_active":"2024-03-04T16:09:30.179819+0000","last_peered":"2024-03-04T16:09:30.179819+0000","last_clean":"2024-03-04T16:09:30.179819+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:09:30.179819+0000","last_undegraded":"2024-03-04T16:09:30.179819+0000","last_fullsized":"2024-03-04T16:09:30.179819+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.732534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":97,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179240+0000","last_change":"2024-03-04T16:08:22.032175+0000","last_active":"2024-03-04T16:09:30.179240+0000","last_peered":"2024-03-04T16:09:30.179240+0000","last_clean":"2024-03-04T16:09:30.179240+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:09:30.179240+0000","last_undegraded":"2024-03-04T16:09:30.179240+0000","last_fullsized":"2024-03-04T16:09:30.179240+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:08:22.032049+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.032049+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:45.673430+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"175'3189","reported_seq":3215,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.402886+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:09:32.402886+0000","last_peered":"2024-03-04T16:09:32.402886+0000","last_clean":"2024-03-04T16:09:32.402886+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:09:32.402886+0000","last_undegraded":"2024-03-04T16:09:32.402886+0000","last_fullsized":"2024-03-04T16:09:32.402886+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.332956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":204096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":232,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.191135+0000","last_change":"2024-03-04T16:06:42.215087+0000","last_active":"2024-03-04T16:09:30.191135+0000","last_peered":"2024-03-04T16:09:30.191135+0000","last_clean":"2024-03-04T16:09:30.191135+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:09:30.191135+0000","last_undegraded":"2024-03-04T16:09:30.191135+0000","last_fullsized":"2024-03-04T16:09:30.191135+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:42.214971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:42.214971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":270,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177239+0000","last_change":"2024-03-04T16:09:01.160778+0000","last_active":"2024-03-04T16:09:30.177239+0000","last_peered":"2024-03-04T16:09:30.177239+0000","last_clean":"2024-03-04T16:09:30.177239+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:09:30.177239+0000","last_undegraded":"2024-03-04T16:09:30.177239+0000","last_fullsized":"2024-03-04T16:09:30.177239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.894027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":272,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176501+0000","last_change":"2024-03-04T16:08:58.764898+0000","last_active":"2024-03-04T16:09:30.176501+0000","last_peered":"2024-03-04T16:09:30.176501+0000","last_clean":"2024-03-04T16:09:30.176501+0000","last_became_active":"2024-03-04T16:07:36.213232+0000","last_became_peered":"2024-03-04T16:07:36.213232+0000","last_unstale":"2024-03-04T16:09:30.176501+0000","last_undegraded":"2024-03-04T16:09:30.176501+0000","last_fullsized":"2024-03-04T16:09:30.176501+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.604580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":378,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185913+0000","last_change":"2024-03-04T16:08:21.819036+0000","last_active":"2024-03-04T16:09:30.185913+0000","last_peered":"2024-03-04T16:09:30.185913+0000","last_clean":"2024-03-04T16:09:30.185913+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:09:30.185913+0000","last_undegraded":"2024-03-04T16:09:30.185913+0000","last_fullsized":"2024-03-04T16:09:30.185913+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.713364+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183033+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:09:30.183033+0000","last_peered":"2024-03-04T16:09:30.183033+0000","last_clean":"2024-03-04T16:09:30.183033+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:09:30.183033+0000","last_undegraded":"2024-03-04T16:09:30.183033+0000","last_fullsized":"2024-03-04T16:09:30.183033+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:59.260044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"175'3378","reported_seq":3404,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.876538+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:09:32.876538+0000","last_peered":"2024-03-04T16:09:32.876538+0000","last_clean":"2024-03-04T16:09:32.876538+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:09:32.876538+0000","last_undegraded":"2024-03-04T16:09:32.876538+0000","last_fullsized":"2024-03-04T16:09:32.876538+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.644896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":216192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":240,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183266+0000","last_change":"2024-03-04T16:09:03.867404+0000","last_active":"2024-03-04T16:09:30.183266+0000","last_peered":"2024-03-04T16:09:30.183266+0000","last_clean":"2024-03-04T16:09:30.183266+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:09:30.183266+0000","last_undegraded":"2024-03-04T16:09:30.183266+0000","last_fullsized":"2024-03-04T16:09:30.183266+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.867298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.867298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.903366+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.184561+0000","last_change":"2024-03-04T16:08:02.744435+0000","last_active":"2024-03-04T16:09:30.184561+0000","last_peered":"2024-03-04T16:09:30.184561+0000","last_clean":"2024-03-04T16:09:30.184561+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:09:30.184561+0000","last_undegraded":"2024-03-04T16:09:30.184561+0000","last_fullsized":"2024-03-04T16:09:30.184561+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:02.744314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.744314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":269,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183141+0000","last_change":"2024-03-04T16:09:21.997899+0000","last_active":"2024-03-04T16:09:30.183141+0000","last_peered":"2024-03-04T16:09:30.183141+0000","last_clean":"2024-03-04T16:09:30.183141+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:09:30.183141+0000","last_undegraded":"2024-03-04T16:09:30.183141+0000","last_fullsized":"2024-03-04T16:09:30.183141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:40.640908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":316,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185384+0000","last_change":"2024-03-04T16:08:26.885868+0000","last_active":"2024-03-04T16:09:30.185384+0000","last_peered":"2024-03-04T16:09:30.185384+0000","last_clean":"2024-03-04T16:09:30.185384+0000","last_became_active":"2024-03-04T16:07:36.224375+0000","last_became_peered":"2024-03-04T16:07:36.224375+0000","last_unstale":"2024-03-04T16:09:30.185384+0000","last_undegraded":"2024-03-04T16:09:30.185384+0000","last_fullsized":"2024-03-04T16:09:30.185384+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:47.549945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186500+0000","last_change":"2024-03-04T16:08:10.648202+0000","last_active":"2024-03-04T16:09:30.186500+0000","last_peered":"2024-03-04T16:09:30.186500+0000","last_clean":"2024-03-04T16:09:30.186500+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:09:30.186500+0000","last_undegraded":"2024-03-04T16:09:30.186500+0000","last_fullsized":"2024-03-04T16:09:30.186500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.648089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.648089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:31.126923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"175'3137","reported_seq":3163,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.422356+0000","last_change":"2024-03-04T16:08:56.231210+0000","last_active":"2024-03-04T16:09:32.422356+0000","last_peered":"2024-03-04T16:09:32.422356+0000","last_clean":"2024-03-04T16:09:32.422356+0000","last_became_active":"2024-03-04T16:08:56.230722+0000","last_became_peered":"2024-03-04T16:08:56.230722+0000","last_unstale":"2024-03-04T16:09:32.422356+0000","last_undegraded":"2024-03-04T16:09:32.422356+0000","last_fullsized":"2024-03-04T16:09:32.422356+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3137,"log_dups_size":0,"ondisk_log_size":3137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.131015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205586432,"num_objects":3137,"num_object_clones":0,"num_object_copies":9411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3137,"num_write_kb":200768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177717+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:09:30.177717+0000","last_peered":"2024-03-04T16:09:30.177717+0000","last_clean":"2024-03-04T16:09:30.177717+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:09:30.177717+0000","last_undegraded":"2024-03-04T16:09:30.177717+0000","last_fullsized":"2024-03-04T16:09:30.177717+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.705424+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":542,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182054+0000","last_change":"2024-03-04T16:09:16.827463+0000","last_active":"2024-03-04T16:09:30.182054+0000","last_peered":"2024-03-04T16:09:30.182054+0000","last_clean":"2024-03-04T16:09:30.182054+0000","last_became_active":"2024-03-04T16:07:39.283273+0000","last_became_peered":"2024-03-04T16:07:39.283273+0000","last_unstale":"2024-03-04T16:09:30.182054+0000","last_undegraded":"2024-03-04T16:09:30.182054+0000","last_fullsized":"2024-03-04T16:09:30.182054+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:09:16.827314+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:09:16.827314+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:46.041701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.187546+0000","last_change":"2024-03-04T16:09:13.880920+0000","last_active":"2024-03-04T16:09:30.187546+0000","last_peered":"2024-03-04T16:09:30.187546+0000","last_clean":"2024-03-04T16:09:30.187546+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:09:30.187546+0000","last_undegraded":"2024-03-04T16:09:30.187546+0000","last_fullsized":"2024-03-04T16:09:30.187546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:33.861869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":360,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183055+0000","last_change":"2024-03-04T16:08:29.900384+0000","last_active":"2024-03-04T16:09:30.183055+0000","last_peered":"2024-03-04T16:09:30.183055+0000","last_clean":"2024-03-04T16:09:30.183055+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:09:30.183055+0000","last_undegraded":"2024-03-04T16:09:30.183055+0000","last_fullsized":"2024-03-04T16:09:30.183055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:08:29.900265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.900265+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:38.478978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182696+0000","last_change":"2024-03-04T16:08:21.802175+0000","last_active":"2024-03-04T16:09:30.182696+0000","last_peered":"2024-03-04T16:09:30.182696+0000","last_clean":"2024-03-04T16:09:30.182696+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:09:30.182696+0000","last_undegraded":"2024-03-04T16:09:30.182696+0000","last_fullsized":"2024-03-04T16:09:30.182696+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:21.802092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.802092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:34.204356+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"175'3164","reported_seq":3190,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.687546+0000","last_change":"2024-03-04T16:08:56.234639+0000","last_active":"2024-03-04T16:09:32.687546+0000","last_peered":"2024-03-04T16:09:32.687546+0000","last_clean":"2024-03-04T16:09:32.687546+0000","last_became_active":"2024-03-04T16:08:56.234353+0000","last_became_peered":"2024-03-04T16:08:56.234353+0000","last_unstale":"2024-03-04T16:09:32.687546+0000","last_undegraded":"2024-03-04T16:09:32.687546+0000","last_fullsized":"2024-03-04T16:09:32.687546+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3164,"log_dups_size":0,"ondisk_log_size":3164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:05.306737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207355904,"num_objects":3164,"num_object_clones":0,"num_object_copies":9492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3164,"num_write_kb":202496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":248,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.184175+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:09:30.184175+0000","last_peered":"2024-03-04T16:09:30.184175+0000","last_clean":"2024-03-04T16:09:30.184175+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:09:30.184175+0000","last_undegraded":"2024-03-04T16:09:30.184175+0000","last_fullsized":"2024-03-04T16:09:30.184175+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.688148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":87,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179649+0000","last_change":"2024-03-04T16:08:10.110344+0000","last_active":"2024-03-04T16:09:30.179649+0000","last_peered":"2024-03-04T16:09:30.179649+0000","last_clean":"2024-03-04T16:09:30.179649+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:09:30.179649+0000","last_undegraded":"2024-03-04T16:09:30.179649+0000","last_fullsized":"2024-03-04T16:09:30.179649+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.109986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.109986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:34.761279+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":675,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.577673+0000","last_change":"2024-03-04T16:09:21.827377+0000","last_active":"2024-03-04T16:09:30.577673+0000","last_peered":"2024-03-04T16:09:30.577673+0000","last_clean":"2024-03-04T16:09:30.577673+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:09:30.577673+0000","last_undegraded":"2024-03-04T16:09:30.577673+0000","last_fullsized":"2024-03-04T16:09:30.577673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.827277+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.759055+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":101,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.189113+0000","last_change":"2024-03-04T16:08:16.126230+0000","last_active":"2024-03-04T16:09:30.189113+0000","last_peered":"2024-03-04T16:09:30.189113+0000","last_clean":"2024-03-04T16:09:30.189113+0000","last_became_active":"2024-03-04T16:07:36.231501+0000","last_became_peered":"2024-03-04T16:07:36.231501+0000","last_unstale":"2024-03-04T16:09:30.189113+0000","last_undegraded":"2024-03-04T16:09:30.189113+0000","last_fullsized":"2024-03-04T16:09:30.189113+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:16.125960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:16.125960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":251,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.178094+0000","last_change":"2024-03-04T16:08:28.262306+0000","last_active":"2024-03-04T16:09:30.178094+0000","last_peered":"2024-03-04T16:09:30.178094+0000","last_clean":"2024-03-04T16:09:30.178094+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:09:30.178094+0000","last_undegraded":"2024-03-04T16:09:30.178094+0000","last_fullsized":"2024-03-04T16:09:30.178094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.262199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.262199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:54.557979+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"175'3187","reported_seq":3213,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.885925+0000","last_change":"2024-03-04T16:08:56.233179+0000","last_active":"2024-03-04T16:09:32.885925+0000","last_peered":"2024-03-04T16:09:32.885925+0000","last_clean":"2024-03-04T16:09:32.885925+0000","last_became_active":"2024-03-04T16:08:56.231149+0000","last_became_peered":"2024-03-04T16:08:56.231149+0000","last_unstale":"2024-03-04T16:09:32.885925+0000","last_undegraded":"2024-03-04T16:09:32.885925+0000","last_fullsized":"2024-03-04T16:09:32.885925+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3187,"log_dups_size":0,"ondisk_log_size":3187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.112092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208863232,"num_objects":3187,"num_object_clones":0,"num_object_copies":9561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3187,"num_write_kb":203968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":240,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186743+0000","last_change":"2024-03-04T16:08:19.795646+0000","last_active":"2024-03-04T16:09:30.186743+0000","last_peered":"2024-03-04T16:09:30.186743+0000","last_clean":"2024-03-04T16:09:30.186743+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:09:30.186743+0000","last_undegraded":"2024-03-04T16:09:30.186743+0000","last_fullsized":"2024-03-04T16:09:30.186743+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.795237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.795237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185821+0000","last_change":"2024-03-04T16:08:24.836558+0000","last_active":"2024-03-04T16:09:30.185821+0000","last_peered":"2024-03-04T16:09:30.185821+0000","last_clean":"2024-03-04T16:09:30.185821+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:09:30.185821+0000","last_undegraded":"2024-03-04T16:09:30.185821+0000","last_fullsized":"2024-03-04T16:09:30.185821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:50.980312+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":219,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.180232+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:09:30.180232+0000","last_peered":"2024-03-04T16:09:30.180232+0000","last_clean":"2024-03-04T16:09:30.180232+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:09:30.180232+0000","last_undegraded":"2024-03-04T16:09:30.180232+0000","last_fullsized":"2024-03-04T16:09:30.180232+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:48.807304+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181575+0000","last_change":"2024-03-04T16:08:06.669836+0000","last_active":"2024-03-04T16:09:30.181575+0000","last_peered":"2024-03-04T16:09:30.181575+0000","last_clean":"2024-03-04T16:09:30.181575+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:09:30.181575+0000","last_undegraded":"2024-03-04T16:09:30.181575+0000","last_fullsized":"2024-03-04T16:09:30.181575+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.669723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.669723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:33.039822+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"175'3261","reported_seq":3287,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.945959+0000","last_change":"2024-03-04T16:08:56.229590+0000","last_active":"2024-03-04T16:09:32.945959+0000","last_peered":"2024-03-04T16:09:32.945959+0000","last_clean":"2024-03-04T16:09:32.945959+0000","last_became_active":"2024-03-04T16:08:56.229291+0000","last_became_peered":"2024-03-04T16:08:56.229291+0000","last_unstale":"2024-03-04T16:09:32.945959+0000","last_undegraded":"2024-03-04T16:09:32.945959+0000","last_fullsized":"2024-03-04T16:09:32.945959+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3261,"log_dups_size":0,"ondisk_log_size":3261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:04.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213712896,"num_objects":3261,"num_object_clones":0,"num_object_copies":9783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3261,"num_write_kb":208704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":248,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.179874+0000","last_change":"2024-03-04T16:09:23.314631+0000","last_active":"2024-03-04T16:09:30.179874+0000","last_peered":"2024-03-04T16:09:30.179874+0000","last_clean":"2024-03-04T16:09:30.179874+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:09:30.179874+0000","last_undegraded":"2024-03-04T16:09:30.179874+0000","last_fullsized":"2024-03-04T16:09:30.179874+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.314502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.314502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:38.273986+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":259,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.178133+0000","last_change":"2024-03-04T16:09:12.013484+0000","last_active":"2024-03-04T16:09:30.178133+0000","last_peered":"2024-03-04T16:09:30.178133+0000","last_clean":"2024-03-04T16:09:30.178133+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:09:30.178133+0000","last_undegraded":"2024-03-04T16:09:30.178133+0000","last_fullsized":"2024-03-04T16:09:30.178133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.570727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":268,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185453+0000","last_change":"2024-03-04T16:08:17.809580+0000","last_active":"2024-03-04T16:09:30.185453+0000","last_peered":"2024-03-04T16:09:30.185453+0000","last_clean":"2024-03-04T16:09:30.185453+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:09:30.185453+0000","last_undegraded":"2024-03-04T16:09:30.185453+0000","last_fullsized":"2024-03-04T16:09:30.185453+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:08:17.809443+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.809443+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"141'10","reported_seq":185,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182097+0000","last_change":"2024-03-04T16:08:13.854540+0000","last_active":"2024-03-04T16:09:30.182097+0000","last_peered":"2024-03-04T16:09:30.182097+0000","last_clean":"2024-03-04T16:09:30.182097+0000","last_became_active":"2024-03-04T16:07:36.219930+0000","last_became_peered":"2024-03-04T16:07:36.219930+0000","last_unstale":"2024-03-04T16:09:30.182097+0000","last_undegraded":"2024-03-04T16:09:30.182097+0000","last_fullsized":"2024-03-04T16:09:30.182097+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:13.854454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.854454+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.923746+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"175'3223","reported_seq":3249,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.376615+0000","last_change":"2024-03-04T16:08:56.789726+0000","last_active":"2024-03-04T16:09:32.376615+0000","last_peered":"2024-03-04T16:09:32.376615+0000","last_clean":"2024-03-04T16:09:32.376615+0000","last_became_active":"2024-03-04T16:08:56.789091+0000","last_became_peered":"2024-03-04T16:08:56.789091+0000","last_unstale":"2024-03-04T16:09:32.376615+0000","last_undegraded":"2024-03-04T16:09:32.376615+0000","last_fullsized":"2024-03-04T16:09:32.376615+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3223,"log_dups_size":0,"ondisk_log_size":3223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.511629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211222528,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3223,"num_write_kb":206272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":248,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.182573+0000","last_change":"2024-03-04T16:09:14.851111+0000","last_active":"2024-03-04T16:09:30.182573+0000","last_peered":"2024-03-04T16:09:30.182573+0000","last_clean":"2024-03-04T16:09:30.182573+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:09:30.182573+0000","last_undegraded":"2024-03-04T16:09:30.182573+0000","last_fullsized":"2024-03-04T16:09:30.182573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:14.851027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:09:14.851027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.585983+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":99,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.189288+0000","last_change":"2024-03-04T16:09:24.277958+0000","last_active":"2024-03-04T16:09:30.189288+0000","last_peered":"2024-03-04T16:09:30.189288+0000","last_clean":"2024-03-04T16:09:30.189288+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:09:30.189288+0000","last_undegraded":"2024-03-04T16:09:30.189288+0000","last_fullsized":"2024-03-04T16:09:30.189288+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:24.277674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:24.277674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:49.480623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":276,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176054+0000","last_change":"2024-03-04T16:08:59.783945+0000","last_active":"2024-03-04T16:09:30.176054+0000","last_peered":"2024-03-04T16:09:30.176054+0000","last_clean":"2024-03-04T16:09:30.176054+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:09:30.176054+0000","last_undegraded":"2024-03-04T16:09:30.176054+0000","last_fullsized":"2024-03-04T16:09:30.176054+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:00.650777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":326,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.188744+0000","last_change":"2024-03-04T16:06:25.283072+0000","last_active":"2024-03-04T16:09:30.188744+0000","last_peered":"2024-03-04T16:09:30.188744+0000","last_clean":"2024-03-04T16:09:30.188744+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:09:30.188744+0000","last_undegraded":"2024-03-04T16:09:30.188744+0000","last_fullsized":"2024-03-04T16:09:30.188744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:06:25.282843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:25.282843+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"175'3366","reported_seq":3392,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.958755+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:09:32.958755+0000","last_peered":"2024-03-04T16:09:32.958755+0000","last_clean":"2024-03-04T16:09:32.958755+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:09:32.958755+0000","last_undegraded":"2024-03-04T16:09:32.958755+0000","last_fullsized":"2024-03-04T16:09:32.958755+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3366,"log_dups_size":0,"ondisk_log_size":3366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.412376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220594176,"num_objects":3366,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":215424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":240,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176028+0000","last_change":"2024-03-04T16:09:05.803480+0000","last_active":"2024-03-04T16:09:30.176028+0000","last_peered":"2024-03-04T16:09:30.176028+0000","last_clean":"2024-03-04T16:09:30.176028+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:09:30.176028+0000","last_undegraded":"2024-03-04T16:09:30.176028+0000","last_fullsized":"2024-03-04T16:09:30.176028+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:30.872236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":256,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.193392+0000","last_change":"2024-03-04T16:07:36.221669+0000","last_active":"2024-03-04T16:09:30.193392+0000","last_peered":"2024-03-04T16:09:30.193392+0000","last_clean":"2024-03-04T16:09:30.193392+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:09:30.193392+0000","last_undegraded":"2024-03-04T16:09:30.193392+0000","last_fullsized":"2024-03-04T16:09:30.193392+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:06:49.140391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":328,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186496+0000","last_change":"2024-03-04T16:08:02.407502+0000","last_active":"2024-03-04T16:09:30.186496+0000","last_peered":"2024-03-04T16:09:30.186496+0000","last_clean":"2024-03-04T16:09:30.186496+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:09:30.186496+0000","last_undegraded":"2024-03-04T16:09:30.186496+0000","last_fullsized":"2024-03-04T16:09:30.186496+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:08:02.407290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.407290+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":244,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.175848+0000","last_change":"2024-03-04T16:07:28.229798+0000","last_active":"2024-03-04T16:09:30.175848+0000","last_peered":"2024-03-04T16:09:30.175848+0000","last_clean":"2024-03-04T16:09:30.175848+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:09:30.175848+0000","last_undegraded":"2024-03-04T16:09:30.175848+0000","last_fullsized":"2024-03-04T16:09:30.175848+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:28.229716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:07:28.229716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"175'3238","reported_seq":3264,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.721779+0000","last_change":"2024-03-04T16:08:56.232940+0000","last_active":"2024-03-04T16:09:32.721779+0000","last_peered":"2024-03-04T16:09:32.721779+0000","last_clean":"2024-03-04T16:09:32.721779+0000","last_became_active":"2024-03-04T16:08:56.232588+0000","last_became_peered":"2024-03-04T16:08:56.232588+0000","last_unstale":"2024-03-04T16:09:32.721779+0000","last_undegraded":"2024-03-04T16:09:32.721779+0000","last_fullsized":"2024-03-04T16:09:32.721779+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3238,"log_dups_size":0,"ondisk_log_size":3238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.367142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212205568,"num_objects":3238,"num_object_clones":0,"num_object_copies":9714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3238,"num_write_kb":207232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":250,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177856+0000","last_change":"2024-03-04T16:08:17.162298+0000","last_active":"2024-03-04T16:09:30.177856+0000","last_peered":"2024-03-04T16:09:30.177856+0000","last_clean":"2024-03-04T16:09:30.177856+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:09:30.177856+0000","last_undegraded":"2024-03-04T16:09:30.177856+0000","last_fullsized":"2024-03-04T16:09:30.177856+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:17.162173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.162173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":334,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177057+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:09:30.177057+0000","last_peered":"2024-03-04T16:09:30.177057+0000","last_clean":"2024-03-04T16:09:30.177057+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:09:30.177057+0000","last_undegraded":"2024-03-04T16:09:30.177057+0000","last_fullsized":"2024-03-04T16:09:30.177057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186401+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:09:30.186401+0000","last_peered":"2024-03-04T16:09:30.186401+0000","last_clean":"2024-03-04T16:09:30.186401+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:09:30.186401+0000","last_undegraded":"2024-03-04T16:09:30.186401+0000","last_fullsized":"2024-03-04T16:09:30.186401+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.366856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":273,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.185241+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:09:30.185241+0000","last_peered":"2024-03-04T16:09:30.185241+0000","last_clean":"2024-03-04T16:09:30.185241+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:09:30.185241+0000","last_undegraded":"2024-03-04T16:09:30.185241+0000","last_fullsized":"2024-03-04T16:09:30.185241+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:37.375921+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"175'3276","reported_seq":3302,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.412565+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:09:32.412565+0000","last_peered":"2024-03-04T16:09:32.412565+0000","last_clean":"2024-03-04T16:09:32.412565+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:09:32.412565+0000","last_undegraded":"2024-03-04T16:09:32.412565+0000","last_fullsized":"2024-03-04T16:09:32.412565+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3276,"log_dups_size":0,"ondisk_log_size":3276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3276,"num_write_kb":209664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":232,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186041+0000","last_change":"2024-03-04T16:08:12.467279+0000","last_active":"2024-03-04T16:09:30.186041+0000","last_peered":"2024-03-04T16:09:30.186041+0000","last_clean":"2024-03-04T16:09:30.186041+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:09:30.186041+0000","last_undegraded":"2024-03-04T16:09:30.186041+0000","last_fullsized":"2024-03-04T16:09:30.186041+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.082303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":250,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.177995+0000","last_change":"2024-03-04T16:09:06.138702+0000","last_active":"2024-03-04T16:09:30.177995+0000","last_peered":"2024-03-04T16:09:30.177995+0000","last_clean":"2024-03-04T16:09:30.177995+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:09:30.177995+0000","last_undegraded":"2024-03-04T16:09:30.177995+0000","last_fullsized":"2024-03-04T16:09:30.177995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.908387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.175883+0000","last_change":"2024-03-04T16:09:06.758717+0000","last_active":"2024-03-04T16:09:30.175883+0000","last_peered":"2024-03-04T16:09:30.175883+0000","last_clean":"2024-03-04T16:09:30.175883+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:09:30.175883+0000","last_undegraded":"2024-03-04T16:09:30.175883+0000","last_fullsized":"2024-03-04T16:09:30.175883+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:27.733168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":272,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176879+0000","last_change":"2024-03-04T16:08:02.022242+0000","last_active":"2024-03-04T16:09:30.176879+0000","last_peered":"2024-03-04T16:09:30.176879+0000","last_clean":"2024-03-04T16:09:30.176879+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:09:30.176879+0000","last_undegraded":"2024-03-04T16:09:30.176879+0000","last_fullsized":"2024-03-04T16:09:30.176879+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:08:02.022098+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.022098+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"175'3236","reported_seq":3261,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.726533+0000","last_change":"2024-03-04T16:08:56.233870+0000","last_active":"2024-03-04T16:09:32.726533+0000","last_peered":"2024-03-04T16:09:32.726533+0000","last_clean":"2024-03-04T16:09:32.726533+0000","last_became_active":"2024-03-04T16:08:56.233555+0000","last_became_peered":"2024-03-04T16:08:56.233555+0000","last_unstale":"2024-03-04T16:09:32.726533+0000","last_undegraded":"2024-03-04T16:09:32.726533+0000","last_fullsized":"2024-03-04T16:09:32.726533+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.436373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212074496,"num_objects":3236,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":207104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":247,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183805+0000","last_change":"2024-03-04T16:08:30.840092+0000","last_active":"2024-03-04T16:09:30.183805+0000","last_peered":"2024-03-04T16:09:30.183805+0000","last_clean":"2024-03-04T16:09:30.183805+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:09:30.183805+0000","last_undegraded":"2024-03-04T16:09:30.183805+0000","last_fullsized":"2024-03-04T16:09:30.183805+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.142723+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":252,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186056+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:09:30.186056+0000","last_peered":"2024-03-04T16:09:30.186056+0000","last_clean":"2024-03-04T16:09:30.186056+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:09:30.186056+0000","last_undegraded":"2024-03-04T16:09:30.186056+0000","last_fullsized":"2024-03-04T16:09:30.186056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:28.310699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":260,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.181667+0000","last_change":"2024-03-04T16:09:22.995174+0000","last_active":"2024-03-04T16:09:30.181667+0000","last_peered":"2024-03-04T16:09:30.181667+0000","last_clean":"2024-03-04T16:09:30.181667+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:09:30.181667+0000","last_undegraded":"2024-03-04T16:09:30.181667+0000","last_fullsized":"2024-03-04T16:09:30.181667+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:39.238258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":276,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176442+0000","last_change":"2024-03-04T16:07:22.316627+0000","last_active":"2024-03-04T16:09:30.176442+0000","last_peered":"2024-03-04T16:09:30.176442+0000","last_clean":"2024-03-04T16:09:30.176442+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:09:30.176442+0000","last_undegraded":"2024-03-04T16:09:30.176442+0000","last_fullsized":"2024-03-04T16:09:30.176442+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:07:22.316461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:22.316461+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"175'3237","reported_seq":3263,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:32.877144+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:09:32.877144+0000","last_peered":"2024-03-04T16:09:32.877144+0000","last_clean":"2024-03-04T16:09:32.877144+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:09:32.877144+0000","last_undegraded":"2024-03-04T16:09:32.877144+0000","last_fullsized":"2024-03-04T16:09:32.877144+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:09.458186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":207168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":239,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176951+0000","last_change":"2024-03-04T16:08:13.787566+0000","last_active":"2024-03-04T16:09:30.176951+0000","last_peered":"2024-03-04T16:09:30.176951+0000","last_clean":"2024-03-04T16:09:30.176951+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:09:30.176951+0000","last_undegraded":"2024-03-04T16:09:30.176951+0000","last_fullsized":"2024-03-04T16:09:30.176951+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":345,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.183610+0000","last_change":"2024-03-04T16:08:28.762450+0000","last_active":"2024-03-04T16:09:30.183610+0000","last_peered":"2024-03-04T16:09:30.183610+0000","last_clean":"2024-03-04T16:09:30.183610+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:09:30.183610+0000","last_undegraded":"2024-03-04T16:09:30.183610+0000","last_fullsized":"2024-03-04T16:09:30.183610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-04T16:08:28.762324+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.762324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.916266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":256,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.186530+0000","last_change":"2024-03-04T16:07:36.224345+0000","last_active":"2024-03-04T16:09:30.186530+0000","last_peered":"2024-03-04T16:09:30.186530+0000","last_clean":"2024-03-04T16:09:30.186530+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:09:30.186530+0000","last_undegraded":"2024-03-04T16:09:30.186530+0000","last_fullsized":"2024-03-04T16:09:30.186530+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:51.144836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:06:51.144836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":263,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-03-04T16:09:30.176840+0000","last_change":"2024-03-04T16:09:08.714773+0000","last_active":"2024-03-04T16:09:30.176840+0000","last_peered":"2024-03-04T16:09:30.176840+0000","last_clean":"2024-03-04T16:09:30.176840+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:09:30.176840+0000","last_undegraded":"2024-03-04T16:09:30.176840+0000","last_fullsized":"2024-03-04T16:09:30.176840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.690427+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3402039296,"num_objects":51911,"num_object_clones":0,"num_object_copies":155733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51911,"num_write_kb":3322304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10185670656,"data_stored":10185670656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51911,"ondisk_log_size":51911,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070550,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193664,"kb_used_data":1009784,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92551488,"statfs":{"total":95995035648,"available":94772723712,"internally_reserved":0,"allocated":1034018816,"data_stored":1033580164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364890,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145388,"kb_used_data":1015648,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92599764,"statfs":{"total":95995035648,"available":94822158336,"internally_reserved":0,"allocated":1040023552,"data_stored":1039607164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1641304,"kb_used_data":1447696,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92103848,"statfs":{"total":95995035648,"available":94314340352,"internally_reserved":0,"allocated":1482440704,"data_stored":1481976172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1413184,"kb_used_data":1233464,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92331968,"statfs":{"total":95995035648,"available":94547935232,"internally_reserved":0,"allocated":1263067136,"data_stored":1262627828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1391468,"kb_used_data":1211744,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92353684,"statfs":{"total":95995035648,"available":94570172416,"internally_reserved":0,"allocated":1240825856,"data_stored":1240409644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1527116,"kb_used_data":1252996,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92218036,"statfs":{"total":95995035648,"available":94431268864,"internally_reserved":0,"allocated":1283067904,"data_stored":1282615874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":280689096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953572,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1152128,"kb_used_data":1030712,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":92593024,"statfs":{"total":95995035648,"available":94815256576,"internally_reserved":0,"allocated":1055449088,"data_stored":1055008570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626528,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815340,"kb_used_data":1631456,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91929812,"statfs":{"total":95995035648,"available":94136127488,"internally_reserved":0,"allocated":1670610944,"data_stored":1670194338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188283334},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1284702208,"data_stored":1284702208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1483931648,"data_stored":1483931648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1263140864,"data_stored":1263140864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1290272768,"data_stored":1290272768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1071513600,"data_stored":1071513600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1675362304,"data_stored":1675362304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1053622272,"data_stored":1053622272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1063124992,"data_stored":1063124992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:09:40.229 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-04T16:09:40.229 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:09:40.229 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-04T16:09:40.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:09:40.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:40.274 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:40.376 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:40.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:40.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:40.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:40.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:40.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:40.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:40.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:40.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:40.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:40.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:41.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 33 16 63804 63788 120.795 87.4375 0.00645911 0.00827317 2024-03-04T16:09:41.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:41.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:41.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:41.267 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:41.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:41.405 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[29779]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[29779]: osdmap e179: 8 total, 8 up, 8 in 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[38113]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:41 smithi038 ceph-mon[38113]: osdmap e179: 8 total, 8 up, 8 in 2024-03-04T16:09:41.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:41.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:41 smithi176 ceph-mon[33619]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:41 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:41 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:41 smithi176 ceph-mon[33619]: osdmap e179: 8 total, 8 up, 8 in 2024-03-04T16:09:41.614 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:41.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:41.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:41.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:41.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:41.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:41.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:42.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:42.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 34 16 65148 65132 119.713 84 0.00544814 0.00834772 2024-03-04T16:09:42.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:42.141 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:42.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:42 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:42.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:42 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:42.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:42.285 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:42.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.441 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:42.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:42.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:42.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:42 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:09:42.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.717 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:42.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:42.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:42.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:42.890 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:42.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:09:42.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:42.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:42.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:43.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 35 16 66482 66466 118.674 83.375 0.0102635 0.0084211 2024-03-04T16:09:43.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:43.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:43.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:43.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:43 smithi038 ceph-mon[29779]: pgmap v292: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:43 smithi038 ceph-mon[38113]: pgmap v292: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:43.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:43.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.423 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:43.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:43.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:43.570 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:43.582 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:43 smithi176 ceph-mon[33619]: pgmap v292: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-04T16:09:43.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:43.706 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:43.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:43.845 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:43.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:43.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:43.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:44.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 36 15 67793 67778 117.655 82 0.00677025 0.0084944 2024-03-04T16:09:44.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.127 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:44.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:44.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:44.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:44.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:44.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:44.420 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:44.420 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[29779]: pgmap v293: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-04T16:09:44.420 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[38113]: pgmap v293: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.421 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:44 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:44.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.529 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:44.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:44 smithi176 ceph-mon[33619]: pgmap v293: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-03-04T16:09:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:09:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:09:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:44 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:09:44.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:44.660 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:44.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.809 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:44.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:44.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:44.945 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:44.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:45.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:45.078 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 37 16 69064 69048 116.621 79.375 0.0217435 0.00856937 2024-03-04T16:09:45.089 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:45.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:45.218 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:45.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:45.349 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:45.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:45.497 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:45.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:45.637 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:45.739 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:45.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:45.783 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:09:45.829 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:09:45.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:45.876 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:09:45.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:45.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:45.923 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:09:45.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:09:46.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:46.016 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:09:46.065 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:46.066 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:09:46.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 38 16 70046 70030 115.167 61.375 0.00510772 0.00867642 2024-03-04T16:09:46.110 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:09:46.114 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:09:46.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:46.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:46.246 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:09:46.246 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:09:46.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:46.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:46.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:46.451 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:46.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:46 smithi038 ceph-mon[29779]: osdmap e180: 8 total, 8 up, 8 in 2024-03-04T16:09:46.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/259506177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:46.463 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:46 smithi038 ceph-mon[38113]: osdmap e180: 8 total, 8 up, 8 in 2024-03-04T16:09:46.463 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/259506177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:46.492 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:46.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:46 smithi176 ceph-mon[33619]: osdmap e180: 8 total, 8 up, 8 in 2024-03-04T16:09:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/259506177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:09:46.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:46.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:46.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:46.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:46.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:46.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:46.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:47.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:47.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:47.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 39 16 71352 71336 114.305 81.625 0.00736029 0.00874312 2024-03-04T16:09:47.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:47.173 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:47.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:47.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:47.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:47.429 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:09:47.442 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:09:47.442 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:47 smithi038 ceph-mon[29779]: pgmap v295: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 681 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-04T16:09:47.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:47 smithi038 ceph-mon[38113]: pgmap v295: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 681 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-04T16:09:47.450 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:47.552 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:47.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:47 smithi176 ceph-mon[33619]: pgmap v295: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 681 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-04T16:09:47.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.732 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:47.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:47.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:47.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:47.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:47.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:47.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:09:48.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:48.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:09:48.079910+0000 min lat: 0.00119552 max lat: 0.122782 avg lat: 0.00881163 2024-03-04T16:09:48.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:09:48.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 40 16 72612 72596 113.416 78.75 0.00620768 0.00881163 2024-03-04T16:09:48.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:48.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:48.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:48.437 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:48.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[29779]: osdmap e181: 8 total, 8 up, 8 in 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[29779]: from='client.15213 v1:172.21.15.38:0/458117667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[29779]: pgmap v297: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:09:48.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:09:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[38113]: osdmap e181: 8 total, 8 up, 8 in 2024-03-04T16:09:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[38113]: from='client.15213 v1:172.21.15.38:0/458117667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:48 smithi038 ceph-mon[38113]: pgmap v297: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-04T16:09:48.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:48 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:09:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:48 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:09:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:48 smithi176 ceph-mon[33619]: osdmap e181: 8 total, 8 up, 8 in 2024-03-04T16:09:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:48 smithi176 ceph-mon[33619]: from='client.15213 v1:172.21.15.38:0/458117667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:09:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:48 smithi176 ceph-mon[33619]: pgmap v297: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-04T16:09:48.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:48.663 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:48.699 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-03-04T16:09:47.209038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4323678558,"num_objects":66163,"num_object_clones":0,"num_object_copies":198489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66163,"num_whiteouts":0,"num_read":1335,"num_read_kb":1181,"num_write":66580,"num_write_kb":4224221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2956616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66623,"ondisk_log_size":66623,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14072176,"kb_used_data":12516380,"kb_used_omap":12,"kb_used_meta":1555699,"kb_avail":735889040,"statfs":{"total":767960285184,"available":753550376960,"internally_reserved":0,"allocated":12816773120,"data_stored":12813116650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":1593036338},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,11,4,15],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":461438976,"num_objects":7041,"num_object_clones":0,"num_object_copies":21123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7041,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7041,"num_write_kb":450624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.782335"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183109+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:09:40.183109+0000","last_peered":"2024-03-04T16:09:40.183109+0000","last_clean":"2024-03-04T16:09:40.183109+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:09:40.183109+0000","last_undegraded":"2024-03-04T16:09:40.183109+0000","last_fullsized":"2024-03-04T16:09:40.183109+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:44.024543+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.203302+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:09:40.203302+0000","last_peered":"2024-03-04T16:09:40.203302+0000","last_clean":"2024-03-04T16:09:40.203302+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:09:40.203302+0000","last_undegraded":"2024-03-04T16:09:40.203302+0000","last_fullsized":"2024-03-04T16:09:40.203302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":96,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193980+0000","last_change":"2024-03-04T16:08:56.062397+0000","last_active":"2024-03-04T16:09:40.193980+0000","last_peered":"2024-03-04T16:09:40.193980+0000","last_clean":"2024-03-04T16:09:40.193980+0000","last_became_active":"2024-03-04T16:07:36.205930+0000","last_became_peered":"2024-03-04T16:07:36.205930+0000","last_unstale":"2024-03-04T16:09:40.193980+0000","last_undegraded":"2024-03-04T16:09:40.193980+0000","last_fullsized":"2024-03-04T16:09:40.193980+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.471716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183058+0000","last_change":"2024-03-04T16:08:15.752830+0000","last_active":"2024-03-04T16:09:40.183058+0000","last_peered":"2024-03-04T16:09:40.183058+0000","last_clean":"2024-03-04T16:09:40.183058+0000","last_became_active":"2024-03-04T16:02:45.747861+0000","last_became_peered":"2024-03-04T16:02:45.747861+0000","last_unstale":"2024-03-04T16:09:40.183058+0000","last_undegraded":"2024-03-04T16:09:40.183058+0000","last_fullsized":"2024-03-04T16:09:40.183058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:15.752737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.752737+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":247,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194888+0000","last_change":"2024-03-04T16:09:20.902067+0000","last_active":"2024-03-04T16:09:40.194888+0000","last_peered":"2024-03-04T16:09:40.194888+0000","last_clean":"2024-03-04T16:09:40.194888+0000","last_became_active":"2024-03-04T16:02:49.792250+0000","last_became_peered":"2024-03-04T16:02:49.792250+0000","last_unstale":"2024-03-04T16:09:40.194888+0000","last_undegraded":"2024-03-04T16:09:40.194888+0000","last_fullsized":"2024-03-04T16:09:40.194888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:34.490944+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183023+0000","last_change":"2024-03-04T16:09:00.760314+0000","last_active":"2024-03-04T16:09:40.183023+0000","last_peered":"2024-03-04T16:09:40.183023+0000","last_clean":"2024-03-04T16:09:40.183023+0000","last_became_active":"2024-03-04T16:02:47.765221+0000","last_became_peered":"2024-03-04T16:02:47.765221+0000","last_unstale":"2024-03-04T16:09:40.183023+0000","last_undegraded":"2024-03-04T16:09:40.183023+0000","last_fullsized":"2024-03-04T16:09:40.183023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.828248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192769+0000","last_change":"2024-03-04T16:08:30.752343+0000","last_active":"2024-03-04T16:09:40.192769+0000","last_peered":"2024-03-04T16:09:40.192769+0000","last_clean":"2024-03-04T16:09:40.192769+0000","last_became_active":"2024-03-04T16:02:43.712476+0000","last_became_peered":"2024-03-04T16:02:43.712476+0000","last_unstale":"2024-03-04T16:09:40.192769+0000","last_undegraded":"2024-03-04T16:09:40.192769+0000","last_fullsized":"2024-03-04T16:09:40.192769+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.745969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":324,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190132+0000","last_change":"2024-03-04T16:08:21.634801+0000","last_active":"2024-03-04T16:09:40.190132+0000","last_peered":"2024-03-04T16:09:40.190132+0000","last_clean":"2024-03-04T16:09:40.190132+0000","last_became_active":"2024-03-04T16:07:36.221904+0000","last_became_peered":"2024-03-04T16:07:36.221904+0000","last_unstale":"2024-03-04T16:09:40.190132+0000","last_undegraded":"2024-03-04T16:09:40.190132+0000","last_fullsized":"2024-03-04T16:09:40.190132+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.436614+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":103,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.189868+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:09:40.189868+0000","last_peered":"2024-03-04T16:09:40.189868+0000","last_clean":"2024-03-04T16:09:40.189868+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:09:40.189868+0000","last_undegraded":"2024-03-04T16:09:40.189868+0000","last_fullsized":"2024-03-04T16:09:40.189868+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:35.559365+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":250,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.188122+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:09:40.188122+0000","last_peered":"2024-03-04T16:09:40.188122+0000","last_clean":"2024-03-04T16:09:40.188122+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:09:40.188122+0000","last_undegraded":"2024-03-04T16:09:40.188122+0000","last_fullsized":"2024-03-04T16:09:40.188122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.564603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187626+0000","last_change":"2024-03-04T16:08:18.150034+0000","last_active":"2024-03-04T16:09:40.187626+0000","last_peered":"2024-03-04T16:09:40.187626+0000","last_clean":"2024-03-04T16:09:40.187626+0000","last_became_active":"2024-03-04T16:02:43.715477+0000","last_became_peered":"2024-03-04T16:02:43.715477+0000","last_unstale":"2024-03-04T16:09:40.187626+0000","last_undegraded":"2024-03-04T16:09:40.187626+0000","last_fullsized":"2024-03-04T16:09:40.187626+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.351993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":307,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183311+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:09:40.183311+0000","last_peered":"2024-03-04T16:09:40.183311+0000","last_clean":"2024-03-04T16:09:40.183311+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:09:40.183311+0000","last_undegraded":"2024-03-04T16:09:40.183311+0000","last_fullsized":"2024-03-04T16:09:40.183311+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187200+0000","last_change":"2024-03-04T16:08:27.210748+0000","last_active":"2024-03-04T16:09:40.187200+0000","last_peered":"2024-03-04T16:09:40.187200+0000","last_clean":"2024-03-04T16:09:40.187200+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:09:40.187200+0000","last_undegraded":"2024-03-04T16:09:40.187200+0000","last_fullsized":"2024-03-04T16:09:40.187200+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.210666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.210666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.351938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192238+0000","last_change":"2024-03-04T16:08:22.797116+0000","last_active":"2024-03-04T16:09:40.192238+0000","last_peered":"2024-03-04T16:09:40.192238+0000","last_clean":"2024-03-04T16:09:40.192238+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:09:40.192238+0000","last_undegraded":"2024-03-04T16:09:40.192238+0000","last_fullsized":"2024-03-04T16:09:40.192238+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.796991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.796991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.136282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187899+0000","last_change":"2024-03-04T16:08:07.153510+0000","last_active":"2024-03-04T16:09:40.187899+0000","last_peered":"2024-03-04T16:09:40.187899+0000","last_clean":"2024-03-04T16:09:40.187899+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:09:40.187899+0000","last_undegraded":"2024-03-04T16:09:40.187899+0000","last_fullsized":"2024-03-04T16:09:40.187899+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:07.153434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.153434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":279,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183481+0000","last_change":"2024-03-04T16:09:04.791187+0000","last_active":"2024-03-04T16:09:40.183481+0000","last_peered":"2024-03-04T16:09:40.183481+0000","last_clean":"2024-03-04T16:09:40.183481+0000","last_became_active":"2024-03-04T16:04:22.732082+0000","last_became_peered":"2024-03-04T16:04:22.732082+0000","last_unstale":"2024-03-04T16:09:40.183481+0000","last_undegraded":"2024-03-04T16:09:40.183481+0000","last_fullsized":"2024-03-04T16:09:40.183481+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:14.516520+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183753+0000","last_change":"2024-03-04T16:09:03.790808+0000","last_active":"2024-03-04T16:09:40.183753+0000","last_peered":"2024-03-04T16:09:40.183753+0000","last_clean":"2024-03-04T16:09:40.183753+0000","last_became_active":"2024-03-04T16:02:49.785421+0000","last_became_peered":"2024-03-04T16:02:49.785421+0000","last_unstale":"2024-03-04T16:09:40.183753+0000","last_undegraded":"2024-03-04T16:09:40.183753+0000","last_fullsized":"2024-03-04T16:09:40.183753+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:13.192635+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.200953+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:09:40.200953+0000","last_peered":"2024-03-04T16:09:40.200953+0000","last_clean":"2024-03-04T16:09:40.200953+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:09:40.200953+0000","last_undegraded":"2024-03-04T16:09:40.200953+0000","last_fullsized":"2024-03-04T16:09:40.200953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.853952+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191125+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:09:40.191125+0000","last_peered":"2024-03-04T16:09:40.191125+0000","last_clean":"2024-03-04T16:09:40.191125+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:09:40.191125+0000","last_undegraded":"2024-03-04T16:09:40.191125+0000","last_fullsized":"2024-03-04T16:09:40.191125+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.718715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":288,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194122+0000","last_change":"2024-03-04T16:08:27.760340+0000","last_active":"2024-03-04T16:09:40.194122+0000","last_peered":"2024-03-04T16:09:40.194122+0000","last_clean":"2024-03-04T16:09:40.194122+0000","last_became_active":"2024-03-04T16:02:45.746492+0000","last_became_peered":"2024-03-04T16:02:45.746492+0000","last_unstale":"2024-03-04T16:09:40.194122+0000","last_undegraded":"2024-03-04T16:09:40.194122+0000","last_fullsized":"2024-03-04T16:09:40.194122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:27.760194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.760194+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.866875+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":240,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.200994+0000","last_change":"2024-03-04T16:06:26.278970+0000","last_active":"2024-03-04T16:09:40.200994+0000","last_peered":"2024-03-04T16:09:40.200994+0000","last_clean":"2024-03-04T16:09:40.200994+0000","last_became_active":"2024-03-04T16:02:49.792668+0000","last_became_peered":"2024-03-04T16:02:49.792668+0000","last_unstale":"2024-03-04T16:09:40.200994+0000","last_undegraded":"2024-03-04T16:09:40.200994+0000","last_fullsized":"2024-03-04T16:09:40.200994+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:26.278824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:26.278824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713817+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183724+0000","last_change":"2024-03-04T16:08:04.014351+0000","last_active":"2024-03-04T16:09:40.183724+0000","last_peered":"2024-03-04T16:09:40.183724+0000","last_clean":"2024-03-04T16:09:40.183724+0000","last_became_active":"2024-03-04T16:02:47.769574+0000","last_became_peered":"2024-03-04T16:02:47.769574+0000","last_unstale":"2024-03-04T16:09:40.183724+0000","last_undegraded":"2024-03-04T16:09:40.183724+0000","last_fullsized":"2024-03-04T16:09:40.183724+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:04.014236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.014236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183323+0000","last_change":"2024-03-04T16:09:23.836557+0000","last_active":"2024-03-04T16:09:40.183323+0000","last_peered":"2024-03-04T16:09:40.183323+0000","last_clean":"2024-03-04T16:09:40.183323+0000","last_became_active":"2024-03-04T16:02:43.708395+0000","last_became_peered":"2024-03-04T16:02:43.708395+0000","last_unstale":"2024-03-04T16:09:40.183323+0000","last_undegraded":"2024-03-04T16:09:40.183323+0000","last_fullsized":"2024-03-04T16:09:40.183323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:43.944821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193022+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:09:40.193022+0000","last_peered":"2024-03-04T16:09:40.193022+0000","last_clean":"2024-03-04T16:09:40.193022+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:09:40.193022+0000","last_undegraded":"2024-03-04T16:09:40.193022+0000","last_fullsized":"2024-03-04T16:09:40.193022+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:43.843527+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":249,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193400+0000","last_change":"2024-03-04T16:09:01.793018+0000","last_active":"2024-03-04T16:09:40.193400+0000","last_peered":"2024-03-04T16:09:40.193400+0000","last_clean":"2024-03-04T16:09:40.193400+0000","last_became_active":"2024-03-04T16:02:49.792392+0000","last_became_peered":"2024-03-04T16:02:49.792392+0000","last_unstale":"2024-03-04T16:09:40.193400+0000","last_undegraded":"2024-03-04T16:09:40.193400+0000","last_fullsized":"2024-03-04T16:09:40.193400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:09:01.792870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.792870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:29.041563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"124'9","reported_seq":356,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193341+0000","last_change":"2024-03-04T16:08:20.850258+0000","last_active":"2024-03-04T16:09:40.193341+0000","last_peered":"2024-03-04T16:09:40.193341+0000","last_clean":"2024-03-04T16:09:40.193341+0000","last_became_active":"2024-03-04T16:02:47.771330+0000","last_became_peered":"2024-03-04T16:02:47.771330+0000","last_unstale":"2024-03-04T16:09:40.193341+0000","last_undegraded":"2024-03-04T16:09:40.193341+0000","last_fullsized":"2024-03-04T16:09:40.193341+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:08:20.850137+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:06:42.058461+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.850137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.862934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.203396+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:09:40.203396+0000","last_peered":"2024-03-04T16:09:40.203396+0000","last_clean":"2024-03-04T16:09:40.203396+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:09:40.203396+0000","last_undegraded":"2024-03-04T16:09:40.203396+0000","last_fullsized":"2024-03-04T16:09:40.203396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":108,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193988+0000","last_change":"2024-03-04T16:08:04.989047+0000","last_active":"2024-03-04T16:09:40.193988+0000","last_peered":"2024-03-04T16:09:40.193988+0000","last_clean":"2024-03-04T16:09:40.193988+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:09:40.193988+0000","last_undegraded":"2024-03-04T16:09:40.193988+0000","last_fullsized":"2024-03-04T16:09:40.193988+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:08:04.988877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:04.988877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":247,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.201105+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:09:40.201105+0000","last_peered":"2024-03-04T16:09:40.201105+0000","last_clean":"2024-03-04T16:09:40.201105+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:09:40.201105+0000","last_undegraded":"2024-03-04T16:09:40.201105+0000","last_fullsized":"2024-03-04T16:09:40.201105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810321+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194233+0000","last_change":"2024-03-04T16:09:01.127663+0000","last_active":"2024-03-04T16:09:40.194233+0000","last_peered":"2024-03-04T16:09:40.194233+0000","last_clean":"2024-03-04T16:09:40.194233+0000","last_became_active":"2024-03-04T16:02:47.775764+0000","last_became_peered":"2024-03-04T16:02:47.775764+0000","last_unstale":"2024-03-04T16:09:40.194233+0000","last_undegraded":"2024-03-04T16:09:40.194233+0000","last_fullsized":"2024-03-04T16:09:40.194233+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:01.883191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.189928+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:09:40.189928+0000","last_peered":"2024-03-04T16:09:40.189928+0000","last_clean":"2024-03-04T16:09:40.189928+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:09:40.189928+0000","last_undegraded":"2024-03-04T16:09:40.189928+0000","last_fullsized":"2024-03-04T16:09:40.189928+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.074160+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":301,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193430+0000","last_change":"2024-03-04T16:08:58.094226+0000","last_active":"2024-03-04T16:09:40.193430+0000","last_peered":"2024-03-04T16:09:40.193430+0000","last_clean":"2024-03-04T16:09:40.193430+0000","last_became_active":"2024-03-04T16:04:22.734590+0000","last_became_peered":"2024-03-04T16:04:22.734590+0000","last_unstale":"2024-03-04T16:09:40.193430+0000","last_undegraded":"2024-03-04T16:09:40.193430+0000","last_fullsized":"2024-03-04T16:09:40.193430+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.639004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192131+0000","last_change":"2024-03-04T16:09:19.779562+0000","last_active":"2024-03-04T16:09:40.192131+0000","last_peered":"2024-03-04T16:09:40.192131+0000","last_clean":"2024-03-04T16:09:40.192131+0000","last_became_active":"2024-03-04T16:02:49.790948+0000","last_became_peered":"2024-03-04T16:02:49.790948+0000","last_unstale":"2024-03-04T16:09:40.192131+0000","last_undegraded":"2024-03-04T16:09:40.192131+0000","last_fullsized":"2024-03-04T16:09:40.192131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:09:19.779373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:27.504868+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193000+0000","last_change":"2024-03-04T16:08:56.035409+0000","last_active":"2024-03-04T16:09:40.193000+0000","last_peered":"2024-03-04T16:09:40.193000+0000","last_clean":"2024-03-04T16:09:40.193000+0000","last_became_active":"2024-03-04T16:02:47.770765+0000","last_became_peered":"2024-03-04T16:02:47.770765+0000","last_unstale":"2024-03-04T16:09:40.193000+0000","last_undegraded":"2024-03-04T16:09:40.193000+0000","last_fullsized":"2024-03-04T16:09:40.193000+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.644040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":95,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193282+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:09:40.193282+0000","last_peered":"2024-03-04T16:09:40.193282+0000","last_clean":"2024-03-04T16:09:40.193282+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:09:40.193282+0000","last_undegraded":"2024-03-04T16:09:40.193282+0000","last_fullsized":"2024-03-04T16:09:40.193282+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.356777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190081+0000","last_change":"2024-03-04T16:06:24.244107+0000","last_active":"2024-03-04T16:09:40.190081+0000","last_peered":"2024-03-04T16:09:40.190081+0000","last_clean":"2024-03-04T16:09:40.190081+0000","last_became_active":"2024-03-04T16:02:45.752342+0000","last_became_peered":"2024-03-04T16:02:45.752342+0000","last_unstale":"2024-03-04T16:09:40.190081+0000","last_undegraded":"2024-03-04T16:09:40.190081+0000","last_fullsized":"2024-03-04T16:09:40.190081+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:06:24.243967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:24.243967+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.852105+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.195081+0000","last_change":"2024-03-04T16:08:06.354795+0000","last_active":"2024-03-04T16:09:40.195081+0000","last_peered":"2024-03-04T16:09:40.195081+0000","last_clean":"2024-03-04T16:09:40.195081+0000","last_became_active":"2024-03-04T16:07:36.224625+0000","last_became_peered":"2024-03-04T16:07:36.224625+0000","last_unstale":"2024-03-04T16:09:40.195081+0000","last_undegraded":"2024-03-04T16:09:40.195081+0000","last_fullsized":"2024-03-04T16:09:40.195081+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714117+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191895+0000","last_change":"2024-03-04T16:08:58.788323+0000","last_active":"2024-03-04T16:09:40.191895+0000","last_peered":"2024-03-04T16:09:40.191895+0000","last_clean":"2024-03-04T16:09:40.191895+0000","last_became_active":"2024-03-04T16:02:47.773492+0000","last_became_peered":"2024-03-04T16:02:47.773492+0000","last_unstale":"2024-03-04T16:09:40.191895+0000","last_undegraded":"2024-03-04T16:09:40.191895+0000","last_fullsized":"2024-03-04T16:09:40.191895+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.788235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.788235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.003335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187457+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:09:40.187457+0000","last_peered":"2024-03-04T16:09:40.187457+0000","last_clean":"2024-03-04T16:09:40.187457+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:09:40.187457+0000","last_undegraded":"2024-03-04T16:09:40.187457+0000","last_fullsized":"2024-03-04T16:09:40.187457+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.344720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190029+0000","last_change":"2024-03-04T16:06:34.283125+0000","last_active":"2024-03-04T16:09:40.190029+0000","last_peered":"2024-03-04T16:09:40.190029+0000","last_clean":"2024-03-04T16:09:40.190029+0000","last_became_active":"2024-03-04T16:02:45.752874+0000","last_became_peered":"2024-03-04T16:02:45.752874+0000","last_unstale":"2024-03-04T16:09:40.190029+0000","last_undegraded":"2024-03-04T16:09:40.190029+0000","last_fullsized":"2024-03-04T16:09:40.190029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:06:34.282977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:06:34.282977+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":95,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.189983+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:09:40.189983+0000","last_peered":"2024-03-04T16:09:40.189983+0000","last_clean":"2024-03-04T16:09:40.189983+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:09:40.189983+0000","last_undegraded":"2024-03-04T16:09:40.189983+0000","last_fullsized":"2024-03-04T16:09:40.189983+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.303288+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.195175+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:09:40.195175+0000","last_peered":"2024-03-04T16:09:40.195175+0000","last_clean":"2024-03-04T16:09:40.195175+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:09:40.195175+0000","last_undegraded":"2024-03-04T16:09:40.195175+0000","last_fullsized":"2024-03-04T16:09:40.195175+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:57.871591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193249+0000","last_change":"2024-03-04T16:09:13.858427+0000","last_active":"2024-03-04T16:09:40.193249+0000","last_peered":"2024-03-04T16:09:40.193249+0000","last_clean":"2024-03-04T16:09:40.193249+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:09:40.193249+0000","last_undegraded":"2024-03-04T16:09:40.193249+0000","last_fullsized":"2024-03-04T16:09:40.193249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.858316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.858316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:16.795714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":283,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.202748+0000","last_change":"2024-03-04T16:09:13.635317+0000","last_active":"2024-03-04T16:09:40.202748+0000","last_peered":"2024-03-04T16:09:40.202748+0000","last_clean":"2024-03-04T16:09:40.202748+0000","last_became_active":"2024-03-04T16:02:45.745318+0000","last_became_peered":"2024-03-04T16:02:45.745318+0000","last_unstale":"2024-03-04T16:09:40.202748+0000","last_undegraded":"2024-03-04T16:09:40.202748+0000","last_fullsized":"2024-03-04T16:09:40.202748+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:31.274560+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":251,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.189695+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:09:40.189695+0000","last_peered":"2024-03-04T16:09:40.189695+0000","last_clean":"2024-03-04T16:09:40.189695+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:09:40.189695+0000","last_undegraded":"2024-03-04T16:09:40.189695+0000","last_fullsized":"2024-03-04T16:09:40.189695+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.202566+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:09:40.202566+0000","last_peered":"2024-03-04T16:09:40.202566+0000","last_clean":"2024-03-04T16:09:40.202566+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:09:40.202566+0000","last_undegraded":"2024-03-04T16:09:40.202566+0000","last_fullsized":"2024-03-04T16:09:40.202566+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:28.419644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183613+0000","last_change":"2024-03-04T16:07:31.213593+0000","last_active":"2024-03-04T16:09:40.183613+0000","last_peered":"2024-03-04T16:09:40.183613+0000","last_clean":"2024-03-04T16:09:40.183613+0000","last_became_active":"2024-03-04T16:02:43.713719+0000","last_became_peered":"2024-03-04T16:02:43.713719+0000","last_unstale":"2024-03-04T16:09:40.183613+0000","last_undegraded":"2024-03-04T16:09:40.183613+0000","last_fullsized":"2024-03-04T16:09:40.183613+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:31.213444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:07:31.213444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183325+0000","last_change":"2024-03-04T16:09:02.785333+0000","last_active":"2024-03-04T16:09:40.183325+0000","last_peered":"2024-03-04T16:09:40.183325+0000","last_clean":"2024-03-04T16:09:40.183325+0000","last_became_active":"2024-03-04T16:02:45.748750+0000","last_became_peered":"2024-03-04T16:02:45.748750+0000","last_unstale":"2024-03-04T16:09:40.183325+0000","last_undegraded":"2024-03-04T16:09:40.183325+0000","last_fullsized":"2024-03-04T16:09:40.183325+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.439534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":129,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192094+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:09:40.192094+0000","last_peered":"2024-03-04T16:09:40.192094+0000","last_clean":"2024-03-04T16:09:40.192094+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:09:40.192094+0000","last_undegraded":"2024-03-04T16:09:40.192094+0000","last_fullsized":"2024-03-04T16:09:40.192094+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:41.311717+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":364,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187721+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:09:40.187721+0000","last_peered":"2024-03-04T16:09:40.187721+0000","last_clean":"2024-03-04T16:09:40.187721+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:09:40.187721+0000","last_undegraded":"2024-03-04T16:09:40.187721+0000","last_fullsized":"2024-03-04T16:09:40.187721+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:43.207576+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192383+0000","last_change":"2024-03-04T16:09:18.820910+0000","last_active":"2024-03-04T16:09:40.192383+0000","last_peered":"2024-03-04T16:09:40.192383+0000","last_clean":"2024-03-04T16:09:40.192383+0000","last_became_active":"2024-03-04T16:02:43.709505+0000","last_became_peered":"2024-03-04T16:02:43.709505+0000","last_unstale":"2024-03-04T16:09:40.192383+0000","last_undegraded":"2024-03-04T16:09:40.192383+0000","last_fullsized":"2024-03-04T16:09:40.192383+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:18.820782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.820782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:44.116052+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192737+0000","last_change":"2024-03-04T16:09:17.965043+0000","last_active":"2024-03-04T16:09:40.192737+0000","last_peered":"2024-03-04T16:09:40.192737+0000","last_clean":"2024-03-04T16:09:40.192737+0000","last_became_active":"2024-03-04T16:04:22.731236+0000","last_became_peered":"2024-03-04T16:04:22.731236+0000","last_unstale":"2024-03-04T16:09:40.192737+0000","last_undegraded":"2024-03-04T16:09:40.192737+0000","last_fullsized":"2024-03-04T16:09:40.192737+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:47.953992+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":247,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183960+0000","last_change":"2024-03-04T16:08:27.876805+0000","last_active":"2024-03-04T16:09:40.183960+0000","last_peered":"2024-03-04T16:09:40.183960+0000","last_clean":"2024-03-04T16:09:40.183960+0000","last_became_active":"2024-03-04T16:02:49.787520+0000","last_became_peered":"2024-03-04T16:02:49.787520+0000","last_unstale":"2024-03-04T16:09:40.183960+0000","last_undegraded":"2024-03-04T16:09:40.183960+0000","last_fullsized":"2024-03-04T16:09:40.183960+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.279216+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"94'9","reported_seq":336,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193197+0000","last_change":"2024-03-04T16:08:07.413971+0000","last_active":"2024-03-04T16:09:40.193197+0000","last_peered":"2024-03-04T16:09:40.193197+0000","last_clean":"2024-03-04T16:09:40.193197+0000","last_became_active":"2024-03-04T16:02:47.773843+0000","last_became_peered":"2024-03-04T16:02:47.773843+0000","last_unstale":"2024-03-04T16:09:40.193197+0000","last_undegraded":"2024-03-04T16:09:40.193197+0000","last_fullsized":"2024-03-04T16:09:40.193197+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":153,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192298+0000","last_change":"2024-03-04T16:08:57.778070+0000","last_active":"2024-03-04T16:09:40.192298+0000","last_peered":"2024-03-04T16:09:40.192298+0000","last_clean":"2024-03-04T16:09:40.192298+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:09:40.192298+0000","last_undegraded":"2024-03-04T16:09:40.192298+0000","last_fullsized":"2024-03-04T16:09:40.192298+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:57.777958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.777958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.367940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187928+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:09:40.187928+0000","last_peered":"2024-03-04T16:09:40.187928+0000","last_clean":"2024-03-04T16:09:40.187928+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:09:40.187928+0000","last_undegraded":"2024-03-04T16:09:40.187928+0000","last_fullsized":"2024-03-04T16:09:40.187928+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.493416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":247,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194411+0000","last_change":"2024-03-04T16:08:12.603101+0000","last_active":"2024-03-04T16:09:40.194411+0000","last_peered":"2024-03-04T16:09:40.194411+0000","last_clean":"2024-03-04T16:09:40.194411+0000","last_became_active":"2024-03-04T16:02:49.789797+0000","last_became_peered":"2024-03-04T16:02:49.789797+0000","last_unstale":"2024-03-04T16:09:40.194411+0000","last_undegraded":"2024-03-04T16:09:40.194411+0000","last_fullsized":"2024-03-04T16:09:40.194411+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.602969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.602969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.866902+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":243,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.200899+0000","last_change":"2024-03-04T16:06:36.235353+0000","last_active":"2024-03-04T16:09:40.200899+0000","last_peered":"2024-03-04T16:09:40.200899+0000","last_clean":"2024-03-04T16:09:40.200899+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:09:40.200899+0000","last_undegraded":"2024-03-04T16:09:40.200899+0000","last_fullsized":"2024-03-04T16:09:40.200899+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:36.235220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:36.235220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713883+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192033+0000","last_change":"2024-03-04T16:09:07.827652+0000","last_active":"2024-03-04T16:09:40.192033+0000","last_peered":"2024-03-04T16:09:40.192033+0000","last_clean":"2024-03-04T16:09:40.192033+0000","last_became_active":"2024-03-04T16:02:43.713760+0000","last_became_peered":"2024-03-04T16:02:43.713760+0000","last_unstale":"2024-03-04T16:09:40.192033+0000","last_undegraded":"2024-03-04T16:09:40.192033+0000","last_fullsized":"2024-03-04T16:09:40.192033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:11.354299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":105,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194080+0000","last_change":"2024-03-04T16:08:20.028471+0000","last_active":"2024-03-04T16:09:40.194080+0000","last_peered":"2024-03-04T16:09:40.194080+0000","last_clean":"2024-03-04T16:09:40.194080+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:09:40.194080+0000","last_undegraded":"2024-03-04T16:09:40.194080+0000","last_fullsized":"2024-03-04T16:09:40.194080+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:20.025566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.025566+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.353163+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":247,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193468+0000","last_change":"2024-03-04T16:08:19.818086+0000","last_active":"2024-03-04T16:09:40.193468+0000","last_peered":"2024-03-04T16:09:40.193468+0000","last_clean":"2024-03-04T16:09:40.193468+0000","last_became_active":"2024-03-04T16:02:49.787320+0000","last_became_peered":"2024-03-04T16:02:49.787320+0000","last_unstale":"2024-03-04T16:09:40.193468+0000","last_undegraded":"2024-03-04T16:09:40.193468+0000","last_fullsized":"2024-03-04T16:09:40.193468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.818005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.818005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.546629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187448+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:09:40.187448+0000","last_peered":"2024-03-04T16:09:40.187448+0000","last_clean":"2024-03-04T16:09:40.187448+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:09:40.187448+0000","last_undegraded":"2024-03-04T16:09:40.187448+0000","last_fullsized":"2024-03-04T16:09:40.187448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:19.621914+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192012+0000","last_change":"2024-03-04T16:08:18.893496+0000","last_active":"2024-03-04T16:09:40.192012+0000","last_peered":"2024-03-04T16:09:40.192012+0000","last_clean":"2024-03-04T16:09:40.192012+0000","last_became_active":"2024-03-04T16:07:36.206947+0000","last_became_peered":"2024-03-04T16:07:36.206947+0000","last_unstale":"2024-03-04T16:09:40.192012+0000","last_undegraded":"2024-03-04T16:09:40.192012+0000","last_fullsized":"2024-03-04T16:09:40.192012+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.893370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.893370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.386307+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.201087+0000","last_change":"2024-03-04T16:06:23.314048+0000","last_active":"2024-03-04T16:09:40.201087+0000","last_peered":"2024-03-04T16:09:40.201087+0000","last_clean":"2024-03-04T16:09:40.201087+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:09:40.201087+0000","last_undegraded":"2024-03-04T16:09:40.201087+0000","last_fullsized":"2024-03-04T16:09:40.201087+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:06:23.313799+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:06:23.313799+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713745+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"179'4068","reported_seq":4101,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.422680+0000","last_change":"2024-03-04T16:08:56.231065+0000","last_active":"2024-03-04T16:09:42.422680+0000","last_peered":"2024-03-04T16:09:42.422680+0000","last_clean":"2024-03-04T16:09:42.422680+0000","last_became_active":"2024-03-04T16:08:56.230573+0000","last_became_peered":"2024-03-04T16:08:56.230573+0000","last_unstale":"2024-03-04T16:09:42.422680+0000","last_undegraded":"2024-03-04T16:09:42.422680+0000","last_fullsized":"2024-03-04T16:09:42.422680+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.913860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266600448,"num_objects":4068,"num_object_clones":0,"num_object_copies":12204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4068,"num_write_kb":260352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187669+0000","last_change":"2024-03-04T16:08:56.044901+0000","last_active":"2024-03-04T16:09:40.187669+0000","last_peered":"2024-03-04T16:09:40.187669+0000","last_clean":"2024-03-04T16:09:40.187669+0000","last_became_active":"2024-03-04T16:02:49.791964+0000","last_became_peered":"2024-03-04T16:02:49.791964+0000","last_unstale":"2024-03-04T16:09:40.187669+0000","last_undegraded":"2024-03-04T16:09:40.187669+0000","last_fullsized":"2024-03-04T16:09:40.187669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.356711+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":296,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192600+0000","last_change":"2024-03-04T16:08:32.803550+0000","last_active":"2024-03-04T16:09:40.192600+0000","last_peered":"2024-03-04T16:09:40.192600+0000","last_clean":"2024-03-04T16:09:40.192600+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:09:40.192600+0000","last_undegraded":"2024-03-04T16:09:40.192600+0000","last_fullsized":"2024-03-04T16:09:40.192600+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:08:32.803433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:32.803433+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:46.264477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":288,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187775+0000","last_change":"2024-03-04T16:08:26.222236+0000","last_active":"2024-03-04T16:09:40.187775+0000","last_peered":"2024-03-04T16:09:40.187775+0000","last_clean":"2024-03-04T16:09:40.187775+0000","last_became_active":"2024-03-04T16:02:45.741318+0000","last_became_peered":"2024-03-04T16:02:45.741318+0000","last_unstale":"2024-03-04T16:09:40.187775+0000","last_undegraded":"2024-03-04T16:09:40.187775+0000","last_fullsized":"2024-03-04T16:09:40.187775+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.781685+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"75'8","reported_seq":344,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192188+0000","last_change":"2024-03-04T16:09:06.869919+0000","last_active":"2024-03-04T16:09:40.192188+0000","last_peered":"2024-03-04T16:09:40.192188+0000","last_clean":"2024-03-04T16:09:40.192188+0000","last_became_active":"2024-03-04T16:02:47.773620+0000","last_became_peered":"2024-03-04T16:02:47.773620+0000","last_unstale":"2024-03-04T16:09:40.192188+0000","last_undegraded":"2024-03-04T16:09:40.192188+0000","last_fullsized":"2024-03-04T16:09:40.192188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:06.869771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.869771+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.331005+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"179'4207","reported_seq":4241,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.440426+0000","last_change":"2024-03-04T16:08:56.789776+0000","last_active":"2024-03-04T16:09:42.440426+0000","last_peered":"2024-03-04T16:09:42.440426+0000","last_clean":"2024-03-04T16:09:42.440426+0000","last_became_active":"2024-03-04T16:08:56.789279+0000","last_became_peered":"2024-03-04T16:08:56.789279+0000","last_unstale":"2024-03-04T16:09:42.440426+0000","last_undegraded":"2024-03-04T16:09:42.440426+0000","last_fullsized":"2024-03-04T16:09:42.440426+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:02.712859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275709952,"num_objects":4207,"num_object_clones":0,"num_object_copies":12621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4207,"num_write_kb":269248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":240,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.194991+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:09:40.194991+0000","last_peered":"2024-03-04T16:09:40.194991+0000","last_clean":"2024-03-04T16:09:40.194991+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:09:40.194991+0000","last_undegraded":"2024-03-04T16:09:40.194991+0000","last_fullsized":"2024-03-04T16:09:40.194991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714086+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192303+0000","last_change":"2024-03-04T16:08:29.793308+0000","last_active":"2024-03-04T16:09:40.192303+0000","last_peered":"2024-03-04T16:09:40.192303+0000","last_clean":"2024-03-04T16:09:40.192303+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:09:40.192303+0000","last_undegraded":"2024-03-04T16:09:40.192303+0000","last_fullsized":"2024-03-04T16:09:40.192303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:29.793217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.793217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.546577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":279,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183541+0000","last_change":"2024-03-04T16:07:24.247724+0000","last_active":"2024-03-04T16:09:40.183541+0000","last_peered":"2024-03-04T16:09:40.183541+0000","last_clean":"2024-03-04T16:09:40.183541+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:09:40.183541+0000","last_undegraded":"2024-03-04T16:09:40.183541+0000","last_fullsized":"2024-03-04T16:09:40.183541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:07:24.247330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:24.247330+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192096+0000","last_change":"2024-03-04T16:09:18.939448+0000","last_active":"2024-03-04T16:09:40.192096+0000","last_peered":"2024-03-04T16:09:40.192096+0000","last_clean":"2024-03-04T16:09:40.192096+0000","last_became_active":"2024-03-04T16:07:36.213031+0000","last_became_peered":"2024-03-04T16:07:36.213031+0000","last_unstale":"2024-03-04T16:09:40.192096+0000","last_undegraded":"2024-03-04T16:09:40.192096+0000","last_fullsized":"2024-03-04T16:09:40.192096+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:18.939310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.939310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:30.070633+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"179'4065","reported_seq":4098,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.962373+0000","last_change":"2024-03-04T16:08:56.789841+0000","last_active":"2024-03-04T16:09:42.962373+0000","last_peered":"2024-03-04T16:09:42.962373+0000","last_clean":"2024-03-04T16:09:42.962373+0000","last_became_active":"2024-03-04T16:08:56.789562+0000","last_became_peered":"2024-03-04T16:08:56.789562+0000","last_unstale":"2024-03-04T16:09:42.962373+0000","last_undegraded":"2024-03-04T16:09:42.962373+0000","last_fullsized":"2024-03-04T16:09:42.962373+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.078343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":260160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191640+0000","last_change":"2024-03-04T16:09:10.830111+0000","last_active":"2024-03-04T16:09:40.191640+0000","last_peered":"2024-03-04T16:09:40.191640+0000","last_clean":"2024-03-04T16:09:40.191640+0000","last_became_active":"2024-03-04T16:02:49.796908+0000","last_became_peered":"2024-03-04T16:02:49.796908+0000","last_unstale":"2024-03-04T16:09:40.191640+0000","last_undegraded":"2024-03-04T16:09:40.191640+0000","last_fullsized":"2024-03-04T16:09:40.191640+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.145188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192385+0000","last_change":"2024-03-04T16:08:22.679372+0000","last_active":"2024-03-04T16:09:40.192385+0000","last_peered":"2024-03-04T16:09:40.192385+0000","last_clean":"2024-03-04T16:09:40.192385+0000","last_became_active":"2024-03-04T16:02:47.773641+0000","last_became_peered":"2024-03-04T16:02:47.773641+0000","last_unstale":"2024-03-04T16:09:40.192385+0000","last_undegraded":"2024-03-04T16:09:40.192385+0000","last_fullsized":"2024-03-04T16:09:40.192385+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.679241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.679241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:49.797013+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193469+0000","last_change":"2024-03-04T16:08:23.747781+0000","last_active":"2024-03-04T16:09:40.193469+0000","last_peered":"2024-03-04T16:09:40.193469+0000","last_clean":"2024-03-04T16:09:40.193469+0000","last_became_active":"2024-03-04T16:02:43.715203+0000","last_became_peered":"2024-03-04T16:02:43.715203+0000","last_unstale":"2024-03-04T16:09:40.193469+0000","last_undegraded":"2024-03-04T16:09:40.193469+0000","last_fullsized":"2024-03-04T16:09:40.193469+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.747649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.747649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.546538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":303,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193037+0000","last_change":"2024-03-04T16:09:23.983219+0000","last_active":"2024-03-04T16:09:40.193037+0000","last_peered":"2024-03-04T16:09:40.193037+0000","last_clean":"2024-03-04T16:09:40.193037+0000","last_became_active":"2024-03-04T16:02:45.746915+0000","last_became_peered":"2024-03-04T16:02:45.746915+0000","last_unstale":"2024-03-04T16:09:40.193037+0000","last_undegraded":"2024-03-04T16:09:40.193037+0000","last_fullsized":"2024-03-04T16:09:40.193037+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.480220+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"179'4125","reported_seq":4157,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.742315+0000","last_change":"2024-03-04T16:08:56.234782+0000","last_active":"2024-03-04T16:09:42.742315+0000","last_peered":"2024-03-04T16:09:42.742315+0000","last_clean":"2024-03-04T16:09:42.742315+0000","last_became_active":"2024-03-04T16:08:56.234486+0000","last_became_peered":"2024-03-04T16:08:56.234486+0000","last_unstale":"2024-03-04T16:09:42.742315+0000","last_undegraded":"2024-03-04T16:09:42.742315+0000","last_fullsized":"2024-03-04T16:09:42.742315+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.872524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336000,"num_objects":4125,"num_object_clones":0,"num_object_copies":12375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":264000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":129,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191935+0000","last_change":"2024-03-04T16:09:15.838949+0000","last_active":"2024-03-04T16:09:40.191935+0000","last_peered":"2024-03-04T16:09:40.191935+0000","last_clean":"2024-03-04T16:09:40.191935+0000","last_became_active":"2024-03-04T16:07:36.211991+0000","last_became_peered":"2024-03-04T16:07:36.211991+0000","last_unstale":"2024-03-04T16:09:40.191935+0000","last_undegraded":"2024-03-04T16:09:40.191935+0000","last_fullsized":"2024-03-04T16:09:40.191935+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.838866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.838866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:40.043588+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192183+0000","last_change":"2024-03-04T16:08:20.664979+0000","last_active":"2024-03-04T16:09:40.192183+0000","last_peered":"2024-03-04T16:09:40.192183+0000","last_clean":"2024-03-04T16:09:40.192183+0000","last_became_active":"2024-03-04T16:02:47.769226+0000","last_became_peered":"2024-03-04T16:02:47.769226+0000","last_unstale":"2024-03-04T16:09:40.192183+0000","last_undegraded":"2024-03-04T16:09:40.192183+0000","last_fullsized":"2024-03-04T16:09:40.192183+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:08:20.664848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.866816+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187962+0000","last_change":"2024-03-04T16:09:04.159447+0000","last_active":"2024-03-04T16:09:40.187962+0000","last_peered":"2024-03-04T16:09:40.187962+0000","last_clean":"2024-03-04T16:09:40.187962+0000","last_became_active":"2024-03-04T16:02:43.713460+0000","last_became_peered":"2024-03-04T16:02:43.713460+0000","last_unstale":"2024-03-04T16:09:40.187962+0000","last_undegraded":"2024-03-04T16:09:40.187962+0000","last_fullsized":"2024-03-04T16:09:40.187962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:07.732534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":105,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193697+0000","last_change":"2024-03-04T16:08:22.032175+0000","last_active":"2024-03-04T16:09:40.193697+0000","last_peered":"2024-03-04T16:09:40.193697+0000","last_clean":"2024-03-04T16:09:40.193697+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:09:40.193697+0000","last_undegraded":"2024-03-04T16:09:40.193697+0000","last_fullsized":"2024-03-04T16:09:40.193697+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:08:22.032049+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.032049+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:45.673430+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"179'4059","reported_seq":4093,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.408291+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:09:42.408291+0000","last_peered":"2024-03-04T16:09:42.408291+0000","last_clean":"2024-03-04T16:09:42.408291+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:09:42.408291+0000","last_undegraded":"2024-03-04T16:09:42.408291+0000","last_fullsized":"2024-03-04T16:09:42.408291+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.332956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266010624,"num_objects":4059,"num_object_clones":0,"num_object_copies":12177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4059,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4059,"num_write_kb":259776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":240,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193007+0000","last_change":"2024-03-04T16:06:42.215087+0000","last_active":"2024-03-04T16:09:40.193007+0000","last_peered":"2024-03-04T16:09:40.193007+0000","last_clean":"2024-03-04T16:09:40.193007+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:09:40.193007+0000","last_undegraded":"2024-03-04T16:09:40.193007+0000","last_fullsized":"2024-03-04T16:09:40.193007+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:42.214971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:06:42.214971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713989+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":278,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187780+0000","last_change":"2024-03-04T16:09:01.160778+0000","last_active":"2024-03-04T16:09:40.187780+0000","last_peered":"2024-03-04T16:09:40.187780+0000","last_clean":"2024-03-04T16:09:40.187780+0000","last_became_active":"2024-03-04T16:02:42.681617+0000","last_became_peered":"2024-03-04T16:02:42.681617+0000","last_unstale":"2024-03-04T16:09:40.187780+0000","last_undegraded":"2024-03-04T16:09:40.187780+0000","last_fullsized":"2024-03-04T16:09:40.187780+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.894027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183183+0000","last_change":"2024-03-04T16:08:58.764898+0000","last_active":"2024-03-04T16:09:40.183183+0000","last_peered":"2024-03-04T16:09:40.183183+0000","last_clean":"2024-03-04T16:09:40.183183+0000","last_became_active":"2024-03-04T16:07:36.213232+0000","last_became_peered":"2024-03-04T16:07:36.213232+0000","last_unstale":"2024-03-04T16:09:40.183183+0000","last_undegraded":"2024-03-04T16:09:40.183183+0000","last_fullsized":"2024-03-04T16:09:40.183183+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.604580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'28","reported_seq":388,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.201683+0000","last_change":"2024-03-04T16:08:21.819036+0000","last_active":"2024-03-04T16:09:40.201683+0000","last_peered":"2024-03-04T16:09:40.201683+0000","last_clean":"2024-03-04T16:09:40.201683+0000","last_became_active":"2024-03-04T16:02:45.753145+0000","last_became_peered":"2024-03-04T16:02:45.753145+0000","last_unstale":"2024-03-04T16:09:40.201683+0000","last_undegraded":"2024-03-04T16:09:40.201683+0000","last_fullsized":"2024-03-04T16:09:40.201683+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.713364+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":96,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190794+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:09:40.190794+0000","last_peered":"2024-03-04T16:09:40.190794+0000","last_clean":"2024-03-04T16:09:40.190794+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:09:40.190794+0000","last_undegraded":"2024-03-04T16:09:40.190794+0000","last_fullsized":"2024-03-04T16:09:40.190794+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:59.260044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"179'4296","reported_seq":4330,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.948476+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:09:42.948476+0000","last_peered":"2024-03-04T16:09:42.948476+0000","last_clean":"2024-03-04T16:09:42.948476+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:09:42.948476+0000","last_undegraded":"2024-03-04T16:09:42.948476+0000","last_fullsized":"2024-03-04T16:09:42.948476+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.644896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281542656,"num_objects":4296,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4296,"num_write_kb":274944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193617+0000","last_change":"2024-03-04T16:09:03.867404+0000","last_active":"2024-03-04T16:09:40.193617+0000","last_peered":"2024-03-04T16:09:40.193617+0000","last_clean":"2024-03-04T16:09:40.193617+0000","last_became_active":"2024-03-04T16:02:49.790773+0000","last_became_peered":"2024-03-04T16:02:49.790773+0000","last_unstale":"2024-03-04T16:09:40.193617+0000","last_undegraded":"2024-03-04T16:09:40.193617+0000","last_fullsized":"2024-03-04T16:09:40.193617+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.867298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.867298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.903366+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.200201+0000","last_change":"2024-03-04T16:08:02.744435+0000","last_active":"2024-03-04T16:09:40.200201+0000","last_peered":"2024-03-04T16:09:40.200201+0000","last_clean":"2024-03-04T16:09:40.200201+0000","last_became_active":"2024-03-04T16:02:43.714525+0000","last_became_peered":"2024-03-04T16:02:43.714525+0000","last_unstale":"2024-03-04T16:09:40.200201+0000","last_undegraded":"2024-03-04T16:09:40.200201+0000","last_fullsized":"2024-03-04T16:09:40.200201+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:02.744314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.744314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.198271+0000","last_change":"2024-03-04T16:09:21.997899+0000","last_active":"2024-03-04T16:09:40.198271+0000","last_peered":"2024-03-04T16:09:40.198271+0000","last_clean":"2024-03-04T16:09:40.198271+0000","last_became_active":"2024-03-04T16:02:42.682797+0000","last_became_peered":"2024-03-04T16:02:42.682797+0000","last_unstale":"2024-03-04T16:09:40.198271+0000","last_undegraded":"2024-03-04T16:09:40.198271+0000","last_fullsized":"2024-03-04T16:09:40.198271+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:40.640908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001794038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":324,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.203346+0000","last_change":"2024-03-04T16:08:26.885868+0000","last_active":"2024-03-04T16:09:40.203346+0000","last_peered":"2024-03-04T16:09:40.203346+0000","last_clean":"2024-03-04T16:09:40.203346+0000","last_became_active":"2024-03-04T16:07:36.224375+0000","last_became_peered":"2024-03-04T16:07:36.224375+0000","last_unstale":"2024-03-04T16:09:40.203346+0000","last_undegraded":"2024-03-04T16:09:40.203346+0000","last_fullsized":"2024-03-04T16:09:40.203346+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:47.549945+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192719+0000","last_change":"2024-03-04T16:08:10.648202+0000","last_active":"2024-03-04T16:09:40.192719+0000","last_peered":"2024-03-04T16:09:40.192719+0000","last_clean":"2024-03-04T16:09:40.192719+0000","last_became_active":"2024-03-04T16:02:47.769325+0000","last_became_peered":"2024-03-04T16:02:47.769325+0000","last_unstale":"2024-03-04T16:09:40.192719+0000","last_undegraded":"2024-03-04T16:09:40.192719+0000","last_fullsized":"2024-03-04T16:09:40.192719+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.648089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.648089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:44.866845+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"179'4003","reported_seq":4037,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.424051+0000","last_change":"2024-03-04T16:08:56.231210+0000","last_active":"2024-03-04T16:09:42.424051+0000","last_peered":"2024-03-04T16:09:42.424051+0000","last_clean":"2024-03-04T16:09:42.424051+0000","last_became_active":"2024-03-04T16:08:56.230722+0000","last_became_peered":"2024-03-04T16:08:56.230722+0000","last_unstale":"2024-03-04T16:09:42.424051+0000","last_undegraded":"2024-03-04T16:09:42.424051+0000","last_fullsized":"2024-03-04T16:09:42.424051+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4003,"log_dups_size":0,"ondisk_log_size":4003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.131015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":12009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4003,"num_write_kb":256192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":255,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187568+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:09:40.187568+0000","last_peered":"2024-03-04T16:09:40.187568+0000","last_clean":"2024-03-04T16:09:40.187568+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:09:40.187568+0000","last_undegraded":"2024-03-04T16:09:40.187568+0000","last_fullsized":"2024-03-04T16:09:40.187568+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:10.705424+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":550,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191741+0000","last_change":"2024-03-04T16:09:16.827463+0000","last_active":"2024-03-04T16:09:40.191741+0000","last_peered":"2024-03-04T16:09:40.191741+0000","last_clean":"2024-03-04T16:09:40.191741+0000","last_became_active":"2024-03-04T16:07:39.283273+0000","last_became_peered":"2024-03-04T16:07:39.283273+0000","last_unstale":"2024-03-04T16:09:40.191741+0000","last_undegraded":"2024-03-04T16:09:40.191741+0000","last_fullsized":"2024-03-04T16:09:40.191741+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:09:16.827314+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:09:16.827314+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:46.041701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190783+0000","last_change":"2024-03-04T16:09:13.880920+0000","last_active":"2024-03-04T16:09:40.190783+0000","last_peered":"2024-03-04T16:09:40.190783+0000","last_clean":"2024-03-04T16:09:40.190783+0000","last_became_active":"2024-03-04T16:02:43.714367+0000","last_became_peered":"2024-03-04T16:02:43.714367+0000","last_unstale":"2024-03-04T16:09:40.190783+0000","last_undegraded":"2024-03-04T16:09:40.190783+0000","last_fullsized":"2024-03-04T16:09:40.190783+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:33.861869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"75'16","reported_seq":370,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.195903+0000","last_change":"2024-03-04T16:08:29.900384+0000","last_active":"2024-03-04T16:09:40.195903+0000","last_peered":"2024-03-04T16:09:40.195903+0000","last_clean":"2024-03-04T16:09:40.195903+0000","last_became_active":"2024-03-04T16:02:45.745316+0000","last_became_peered":"2024-03-04T16:02:45.745316+0000","last_unstale":"2024-03-04T16:09:40.195903+0000","last_undegraded":"2024-03-04T16:09:40.195903+0000","last_fullsized":"2024-03-04T16:09:40.195903+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:08:29.900265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:29.900265+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193507+0000","last_change":"2024-03-04T16:08:21.802175+0000","last_active":"2024-03-04T16:09:40.193507+0000","last_peered":"2024-03-04T16:09:40.193507+0000","last_clean":"2024-03-04T16:09:40.193507+0000","last_became_active":"2024-03-04T16:02:47.769169+0000","last_became_peered":"2024-03-04T16:02:47.769169+0000","last_unstale":"2024-03-04T16:09:40.193507+0000","last_undegraded":"2024-03-04T16:09:40.193507+0000","last_fullsized":"2024-03-04T16:09:40.193507+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:21.802092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.802092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.546602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"179'4077","reported_seq":4111,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.715696+0000","last_change":"2024-03-04T16:08:56.234639+0000","last_active":"2024-03-04T16:09:42.715696+0000","last_peered":"2024-03-04T16:09:42.715696+0000","last_clean":"2024-03-04T16:09:42.715696+0000","last_became_active":"2024-03-04T16:08:56.234353+0000","last_became_peered":"2024-03-04T16:08:56.234353+0000","last_unstale":"2024-03-04T16:09:42.715696+0000","last_undegraded":"2024-03-04T16:09:42.715696+0000","last_fullsized":"2024-03-04T16:09:42.715696+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:05.306737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267190272,"num_objects":4077,"num_object_clones":0,"num_object_copies":12231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4077,"num_write_kb":260928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.199351+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:09:40.199351+0000","last_peered":"2024-03-04T16:09:40.199351+0000","last_clean":"2024-03-04T16:09:40.199351+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:09:40.199351+0000","last_undegraded":"2024-03-04T16:09:40.199351+0000","last_fullsized":"2024-03-04T16:09:40.199351+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.688148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":96,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.195020+0000","last_change":"2024-03-04T16:08:10.110344+0000","last_active":"2024-03-04T16:09:40.195020+0000","last_peered":"2024-03-04T16:09:40.195020+0000","last_clean":"2024-03-04T16:09:40.195020+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:09:40.195020+0000","last_undegraded":"2024-03-04T16:09:40.195020+0000","last_fullsized":"2024-03-04T16:09:40.195020+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:10.109986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:10.109986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:45.398082+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":693,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.589660+0000","last_change":"2024-03-04T16:09:21.827377+0000","last_active":"2024-03-04T16:09:40.589660+0000","last_peered":"2024-03-04T16:09:40.589660+0000","last_clean":"2024-03-04T16:09:40.589660+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:09:40.589660+0000","last_undegraded":"2024-03-04T16:09:40.589660+0000","last_fullsized":"2024-03-04T16:09:40.589660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.827277+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:23.759055+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":412,"num_read_kb":407,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":109,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191079+0000","last_change":"2024-03-04T16:08:16.126230+0000","last_active":"2024-03-04T16:09:40.191079+0000","last_peered":"2024-03-04T16:09:40.191079+0000","last_clean":"2024-03-04T16:09:40.191079+0000","last_became_active":"2024-03-04T16:07:36.231501+0000","last_became_peered":"2024-03-04T16:07:36.231501+0000","last_unstale":"2024-03-04T16:09:40.191079+0000","last_undegraded":"2024-03-04T16:09:40.191079+0000","last_fullsized":"2024-03-04T16:09:40.191079+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:16.125960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:16.125960+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187925+0000","last_change":"2024-03-04T16:08:28.262306+0000","last_active":"2024-03-04T16:09:40.187925+0000","last_peered":"2024-03-04T16:09:40.187925+0000","last_clean":"2024-03-04T16:09:40.187925+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:09:40.187925+0000","last_undegraded":"2024-03-04T16:09:40.187925+0000","last_fullsized":"2024-03-04T16:09:40.187925+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.262199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.262199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:54.557979+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"179'4070","reported_seq":4104,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.945326+0000","last_change":"2024-03-04T16:08:56.233179+0000","last_active":"2024-03-04T16:09:42.945326+0000","last_peered":"2024-03-04T16:09:42.945326+0000","last_clean":"2024-03-04T16:09:42.945326+0000","last_became_active":"2024-03-04T16:08:56.231149+0000","last_became_peered":"2024-03-04T16:08:56.231149+0000","last_unstale":"2024-03-04T16:09:42.945326+0000","last_undegraded":"2024-03-04T16:09:42.945326+0000","last_fullsized":"2024-03-04T16:09:42.945326+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:22.112092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266731520,"num_objects":4070,"num_object_clones":0,"num_object_copies":12210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4070,"num_write_kb":260480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.203212+0000","last_change":"2024-03-04T16:08:19.795646+0000","last_active":"2024-03-04T16:09:40.203212+0000","last_peered":"2024-03-04T16:09:40.203212+0000","last_clean":"2024-03-04T16:09:40.203212+0000","last_became_active":"2024-03-04T16:02:49.784922+0000","last_became_peered":"2024-03-04T16:02:49.784922+0000","last_unstale":"2024-03-04T16:09:40.203212+0000","last_undegraded":"2024-03-04T16:09:40.203212+0000","last_fullsized":"2024-03-04T16:09:40.203212+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:19.795237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:19.795237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810226+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.201597+0000","last_change":"2024-03-04T16:08:24.836558+0000","last_active":"2024-03-04T16:09:40.201597+0000","last_peered":"2024-03-04T16:09:40.201597+0000","last_clean":"2024-03-04T16:09:40.201597+0000","last_became_active":"2024-03-04T16:02:43.718888+0000","last_became_peered":"2024-03-04T16:02:43.718888+0000","last_unstale":"2024-03-04T16:09:40.201597+0000","last_undegraded":"2024-03-04T16:09:40.201597+0000","last_fullsized":"2024-03-04T16:09:40.201597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:50.980312+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":227,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.188123+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:09:40.188123+0000","last_peered":"2024-03-04T16:09:40.188123+0000","last_clean":"2024-03-04T16:09:40.188123+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:09:40.188123+0000","last_undegraded":"2024-03-04T16:09:40.188123+0000","last_fullsized":"2024-03-04T16:09:40.188123+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:48.807304+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193238+0000","last_change":"2024-03-04T16:08:06.669836+0000","last_active":"2024-03-04T16:09:40.193238+0000","last_peered":"2024-03-04T16:09:40.193238+0000","last_clean":"2024-03-04T16:09:40.193238+0000","last_became_active":"2024-03-04T16:02:47.767471+0000","last_became_peered":"2024-03-04T16:02:47.767471+0000","last_unstale":"2024-03-04T16:09:40.193238+0000","last_undegraded":"2024-03-04T16:09:40.193238+0000","last_fullsized":"2024-03-04T16:09:40.193238+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.669723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.669723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"179'4147","reported_seq":4181,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.965937+0000","last_change":"2024-03-04T16:08:56.229590+0000","last_active":"2024-03-04T16:09:42.965937+0000","last_peered":"2024-03-04T16:09:42.965937+0000","last_clean":"2024-03-04T16:09:42.965937+0000","last_became_active":"2024-03-04T16:08:56.229291+0000","last_became_peered":"2024-03-04T16:08:56.229291+0000","last_unstale":"2024-03-04T16:09:42.965937+0000","last_undegraded":"2024-03-04T16:09:42.965937+0000","last_fullsized":"2024-03-04T16:09:42.965937+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:04.297081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271777792,"num_objects":4147,"num_object_clones":0,"num_object_copies":12441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4147,"num_write_kb":265408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190079+0000","last_change":"2024-03-04T16:09:23.314631+0000","last_active":"2024-03-04T16:09:40.190079+0000","last_peered":"2024-03-04T16:09:40.190079+0000","last_clean":"2024-03-04T16:09:40.190079+0000","last_became_active":"2024-03-04T16:02:49.796981+0000","last_became_peered":"2024-03-04T16:02:49.796981+0000","last_unstale":"2024-03-04T16:09:40.190079+0000","last_undegraded":"2024-03-04T16:09:40.190079+0000","last_fullsized":"2024-03-04T16:09:40.190079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.314502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.314502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:38.273986+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":267,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187864+0000","last_change":"2024-03-04T16:09:12.013484+0000","last_active":"2024-03-04T16:09:40.187864+0000","last_peered":"2024-03-04T16:09:40.187864+0000","last_clean":"2024-03-04T16:09:40.187864+0000","last_became_active":"2024-03-04T16:02:43.710940+0000","last_became_peered":"2024-03-04T16:02:43.710940+0000","last_unstale":"2024-03-04T16:09:40.187864+0000","last_undegraded":"2024-03-04T16:09:40.187864+0000","last_fullsized":"2024-03-04T16:09:40.187864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:25.570727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":276,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.202648+0000","last_change":"2024-03-04T16:08:17.809580+0000","last_active":"2024-03-04T16:09:40.202648+0000","last_peered":"2024-03-04T16:09:40.202648+0000","last_clean":"2024-03-04T16:09:40.202648+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:09:40.202648+0000","last_undegraded":"2024-03-04T16:09:40.202648+0000","last_fullsized":"2024-03-04T16:09:40.202648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:08:17.809443+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.809443+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.810164+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"141'10","reported_seq":195,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193433+0000","last_change":"2024-03-04T16:08:13.854540+0000","last_active":"2024-03-04T16:09:40.193433+0000","last_peered":"2024-03-04T16:09:40.193433+0000","last_clean":"2024-03-04T16:09:40.193433+0000","last_became_active":"2024-03-04T16:07:36.219930+0000","last_became_peered":"2024-03-04T16:07:36.219930+0000","last_unstale":"2024-03-04T16:09:40.193433+0000","last_undegraded":"2024-03-04T16:09:40.193433+0000","last_fullsized":"2024-03-04T16:09:40.193433+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:13.854454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.854454+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.923746+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.d","version":"179'4148","reported_seq":4181,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.417151+0000","last_change":"2024-03-04T16:08:56.789726+0000","last_active":"2024-03-04T16:09:42.417151+0000","last_peered":"2024-03-04T16:09:42.417151+0000","last_clean":"2024-03-04T16:09:42.417151+0000","last_became_active":"2024-03-04T16:08:56.789091+0000","last_became_peered":"2024-03-04T16:08:56.789091+0000","last_unstale":"2024-03-04T16:09:42.417151+0000","last_undegraded":"2024-03-04T16:09:42.417151+0000","last_fullsized":"2024-03-04T16:09:42.417151+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:24.511629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271843328,"num_objects":4148,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":265472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.192583+0000","last_change":"2024-03-04T16:09:14.851111+0000","last_active":"2024-03-04T16:09:40.192583+0000","last_peered":"2024-03-04T16:09:40.192583+0000","last_clean":"2024-03-04T16:09:40.192583+0000","last_became_active":"2024-03-04T16:02:49.794547+0000","last_became_peered":"2024-03-04T16:02:49.794547+0000","last_unstale":"2024-03-04T16:09:40.192583+0000","last_undegraded":"2024-03-04T16:09:40.192583+0000","last_fullsized":"2024-03-04T16:09:40.192583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:14.851027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:09:14.851027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.585983+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":107,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193883+0000","last_change":"2024-03-04T16:09:24.277958+0000","last_active":"2024-03-04T16:09:40.193883+0000","last_peered":"2024-03-04T16:09:40.193883+0000","last_clean":"2024-03-04T16:09:40.193883+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:09:40.193883+0000","last_undegraded":"2024-03-04T16:09:40.193883+0000","last_fullsized":"2024-03-04T16:09:40.193883+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:24.277674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:24.277674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:49.480623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183377+0000","last_change":"2024-03-04T16:08:59.783945+0000","last_active":"2024-03-04T16:09:40.183377+0000","last_peered":"2024-03-04T16:09:40.183377+0000","last_clean":"2024-03-04T16:09:40.183377+0000","last_became_active":"2024-03-04T16:02:45.739475+0000","last_became_peered":"2024-03-04T16:02:45.739475+0000","last_unstale":"2024-03-04T16:09:40.183377+0000","last_undegraded":"2024-03-04T16:09:40.183377+0000","last_fullsized":"2024-03-04T16:09:40.183377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:00.650777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":336,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190345+0000","last_change":"2024-03-04T16:06:25.283072+0000","last_active":"2024-03-04T16:09:40.190345+0000","last_peered":"2024-03-04T16:09:40.190345+0000","last_clean":"2024-03-04T16:09:40.190345+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:09:40.190345+0000","last_undegraded":"2024-03-04T16:09:40.190345+0000","last_fullsized":"2024-03-04T16:09:40.190345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:06:25.282843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:06:25.282843+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713853+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"179'4250","reported_seq":4284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.964101+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:09:42.964101+0000","last_peered":"2024-03-04T16:09:42.964101+0000","last_clean":"2024-03-04T16:09:42.964101+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:09:42.964101+0000","last_undegraded":"2024-03-04T16:09:42.964101+0000","last_fullsized":"2024-03-04T16:09:42.964101+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:56.412376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528000,"num_objects":4250,"num_object_clones":0,"num_object_copies":12750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":272000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183202+0000","last_change":"2024-03-04T16:09:05.803480+0000","last_active":"2024-03-04T16:09:40.183202+0000","last_peered":"2024-03-04T16:09:40.183202+0000","last_clean":"2024-03-04T16:09:40.183202+0000","last_became_active":"2024-03-04T16:02:49.789093+0000","last_became_peered":"2024-03-04T16:02:49.789093+0000","last_unstale":"2024-03-04T16:09:40.183202+0000","last_undegraded":"2024-03-04T16:09:40.183202+0000","last_fullsized":"2024-03-04T16:09:40.183202+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:30.872236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.200698+0000","last_change":"2024-03-04T16:07:36.221669+0000","last_active":"2024-03-04T16:09:40.200698+0000","last_peered":"2024-03-04T16:09:40.200698+0000","last_clean":"2024-03-04T16:09:40.200698+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:09:40.200698+0000","last_undegraded":"2024-03-04T16:09:40.200698+0000","last_fullsized":"2024-03-04T16:09:40.200698+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:06:49.140391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714019+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":336,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190217+0000","last_change":"2024-03-04T16:08:02.407502+0000","last_active":"2024-03-04T16:09:40.190217+0000","last_peered":"2024-03-04T16:09:40.190217+0000","last_clean":"2024-03-04T16:09:40.190217+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:09:40.190217+0000","last_undegraded":"2024-03-04T16:09:40.190217+0000","last_fullsized":"2024-03-04T16:09:40.190217+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:08:02.407290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.407290+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.714054+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183050+0000","last_change":"2024-03-04T16:07:28.229798+0000","last_active":"2024-03-04T16:09:40.183050+0000","last_peered":"2024-03-04T16:09:40.183050+0000","last_clean":"2024-03-04T16:09:40.183050+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:09:40.183050+0000","last_undegraded":"2024-03-04T16:09:40.183050+0000","last_fullsized":"2024-03-04T16:09:40.183050+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:07:28.229716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:07:28.229716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"179'4088","reported_seq":4122,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.700475+0000","last_change":"2024-03-04T16:08:56.232940+0000","last_active":"2024-03-04T16:09:42.700475+0000","last_peered":"2024-03-04T16:09:42.700475+0000","last_clean":"2024-03-04T16:09:42.700475+0000","last_became_active":"2024-03-04T16:08:56.232588+0000","last_became_peered":"2024-03-04T16:08:56.232588+0000","last_unstale":"2024-03-04T16:09:42.700475+0000","last_undegraded":"2024-03-04T16:09:42.700475+0000","last_fullsized":"2024-03-04T16:09:42.700475+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:08.367142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267911168,"num_objects":4088,"num_object_clones":0,"num_object_copies":12264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4088,"num_write_kb":261632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187527+0000","last_change":"2024-03-04T16:08:17.162298+0000","last_active":"2024-03-04T16:09:40.187527+0000","last_peered":"2024-03-04T16:09:40.187527+0000","last_clean":"2024-03-04T16:09:40.187527+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:09:40.187527+0000","last_undegraded":"2024-03-04T16:09:40.187527+0000","last_fullsized":"2024-03-04T16:09:40.187527+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:17.162173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.162173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:40.351967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":344,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183823+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:09:40.183823+0000","last_peered":"2024-03-04T16:09:40.183823+0000","last_clean":"2024-03-04T16:09:40.183823+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:09:40.183823+0000","last_undegraded":"2024-03-04T16:09:40.183823+0000","last_fullsized":"2024-03-04T16:09:40.183823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.190131+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:09:40.190131+0000","last_peered":"2024-03-04T16:09:40.190131+0000","last_clean":"2024-03-04T16:09:40.190131+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:09:40.190131+0000","last_undegraded":"2024-03-04T16:09:40.190131+0000","last_fullsized":"2024-03-04T16:09:40.190131+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:21.366856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.203306+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:09:40.203306+0000","last_peered":"2024-03-04T16:09:40.203306+0000","last_clean":"2024-03-04T16:09:40.203306+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:09:40.203306+0000","last_undegraded":"2024-03-04T16:09:40.203306+0000","last_fullsized":"2024-03-04T16:09:40.203306+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:42.800493+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"179'4129","reported_seq":4163,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.402689+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:09:42.402689+0000","last_peered":"2024-03-04T16:09:42.402689+0000","last_clean":"2024-03-04T16:09:42.402689+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:09:42.402689+0000","last_undegraded":"2024-03-04T16:09:42.402689+0000","last_fullsized":"2024-03-04T16:09:42.402689+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:17.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270598144,"num_objects":4129,"num_object_clones":0,"num_object_copies":12387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4129,"num_write_kb":264256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":241,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191263+0000","last_change":"2024-03-04T16:08:12.467279+0000","last_active":"2024-03-04T16:09:40.191263+0000","last_peered":"2024-03-04T16:09:40.191263+0000","last_clean":"2024-03-04T16:09:40.191263+0000","last_became_active":"2024-03-04T16:02:49.793685+0000","last_became_peered":"2024-03-04T16:02:49.793685+0000","last_unstale":"2024-03-04T16:09:40.191263+0000","last_undegraded":"2024-03-04T16:09:40.191263+0000","last_fullsized":"2024-03-04T16:09:40.191263+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.082303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.187880+0000","last_change":"2024-03-04T16:09:06.138702+0000","last_active":"2024-03-04T16:09:40.187880+0000","last_peered":"2024-03-04T16:09:40.187880+0000","last_clean":"2024-03-04T16:09:40.187880+0000","last_became_active":"2024-03-04T16:02:47.765722+0000","last_became_peered":"2024-03-04T16:02:47.765722+0000","last_unstale":"2024-03-04T16:09:40.187880+0000","last_undegraded":"2024-03-04T16:09:40.187880+0000","last_fullsized":"2024-03-04T16:09:40.187880+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:20.908387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183054+0000","last_change":"2024-03-04T16:09:06.758717+0000","last_active":"2024-03-04T16:09:40.183054+0000","last_peered":"2024-03-04T16:09:40.183054+0000","last_clean":"2024-03-04T16:09:40.183054+0000","last_became_active":"2024-03-04T16:02:43.705044+0000","last_became_peered":"2024-03-04T16:02:43.705044+0000","last_unstale":"2024-03-04T16:09:40.183054+0000","last_undegraded":"2024-03-04T16:09:40.183054+0000","last_fullsized":"2024-03-04T16:09:40.183054+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:27.733168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183719+0000","last_change":"2024-03-04T16:08:02.022242+0000","last_active":"2024-03-04T16:09:40.183719+0000","last_peered":"2024-03-04T16:09:40.183719+0000","last_clean":"2024-03-04T16:09:40.183719+0000","last_became_active":"2024-03-04T16:02:45.744804+0000","last_became_peered":"2024-03-04T16:02:45.744804+0000","last_unstale":"2024-03-04T16:09:40.183719+0000","last_undegraded":"2024-03-04T16:09:40.183719+0000","last_fullsized":"2024-03-04T16:09:40.183719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:08:02.022098+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:08:02.022098+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"179'4125","reported_seq":4159,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.704544+0000","last_change":"2024-03-04T16:08:56.233870+0000","last_active":"2024-03-04T16:09:42.704544+0000","last_peered":"2024-03-04T16:09:42.704544+0000","last_clean":"2024-03-04T16:09:42.704544+0000","last_became_active":"2024-03-04T16:08:56.233555+0000","last_became_peered":"2024-03-04T16:08:56.233555+0000","last_unstale":"2024-03-04T16:09:42.704544+0000","last_undegraded":"2024-03-04T16:09:42.704544+0000","last_fullsized":"2024-03-04T16:09:42.704544+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4125,"log_dups_size":0,"ondisk_log_size":4125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:55.436373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270336000,"num_objects":4125,"num_object_clones":0,"num_object_copies":12375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4125,"num_write_kb":264000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.196077+0000","last_change":"2024-03-04T16:08:30.840092+0000","last_active":"2024-03-04T16:09:40.196077+0000","last_peered":"2024-03-04T16:09:40.196077+0000","last_clean":"2024-03-04T16:09:40.196077+0000","last_became_active":"2024-03-04T16:02:49.788839+0000","last_became_peered":"2024-03-04T16:02:49.788839+0000","last_unstale":"2024-03-04T16:09:40.196077+0000","last_undegraded":"2024-03-04T16:09:40.196077+0000","last_fullsized":"2024-03-04T16:09:40.196077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.142723+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.201776+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:09:40.201776+0000","last_peered":"2024-03-04T16:09:40.201776+0000","last_clean":"2024-03-04T16:09:40.201776+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:09:40.201776+0000","last_undegraded":"2024-03-04T16:09:40.201776+0000","last_fullsized":"2024-03-04T16:09:40.201776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:28.310699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193334+0000","last_change":"2024-03-04T16:09:22.995174+0000","last_active":"2024-03-04T16:09:40.193334+0000","last_peered":"2024-03-04T16:09:40.193334+0000","last_clean":"2024-03-04T16:09:40.193334+0000","last_became_active":"2024-03-04T16:02:43.715936+0000","last_became_peered":"2024-03-04T16:02:43.715936+0000","last_unstale":"2024-03-04T16:09:40.193334+0000","last_undegraded":"2024-03-04T16:09:40.193334+0000","last_fullsized":"2024-03-04T16:09:40.193334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:39.238258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183603+0000","last_change":"2024-03-04T16:07:22.316627+0000","last_active":"2024-03-04T16:09:40.183603+0000","last_peered":"2024-03-04T16:09:40.183603+0000","last_clean":"2024-03-04T16:09:40.183603+0000","last_became_active":"2024-03-04T16:02:45.743443+0000","last_became_peered":"2024-03-04T16:02:45.743443+0000","last_unstale":"2024-03-04T16:09:40.183603+0000","last_undegraded":"2024-03-04T16:09:40.183603+0000","last_fullsized":"2024-03-04T16:09:40.183603+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:07:22.316461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:07:22.316461+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"179'4108","reported_seq":4142,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:42.949910+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:09:42.949910+0000","last_peered":"2024-03-04T16:09:42.949910+0000","last_clean":"2024-03-04T16:09:42.949910+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:09:42.949910+0000","last_undegraded":"2024-03-04T16:09:42.949910+0000","last_fullsized":"2024-03-04T16:09:42.949910+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4108,"log_dups_size":0,"ondisk_log_size":4108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:09.458186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269221888,"num_objects":4108,"num_object_clones":0,"num_object_copies":12324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4108,"num_write_kb":262912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":248,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183859+0000","last_change":"2024-03-04T16:08:13.787566+0000","last_active":"2024-03-04T16:09:40.183859+0000","last_peered":"2024-03-04T16:09:40.183859+0000","last_clean":"2024-03-04T16:09:40.183859+0000","last_became_active":"2024-03-04T16:02:49.784970+0000","last_became_peered":"2024-03-04T16:02:49.784970+0000","last_unstale":"2024-03-04T16:09:40.183859+0000","last_undegraded":"2024-03-04T16:09:40.183859+0000","last_fullsized":"2024-03-04T16:09:40.183859+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"90'9","reported_seq":356,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.193602+0000","last_change":"2024-03-04T16:08:28.762450+0000","last_active":"2024-03-04T16:09:40.193602+0000","last_peered":"2024-03-04T16:09:40.193602+0000","last_clean":"2024-03-04T16:09:40.193602+0000","last_became_active":"2024-03-04T16:02:47.769251+0000","last_became_peered":"2024-03-04T16:02:47.769251+0000","last_unstale":"2024-03-04T16:09:40.193602+0000","last_undegraded":"2024-03-04T16:09:40.193602+0000","last_fullsized":"2024-03-04T16:09:40.193602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-04T16:08:28.762324+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.762324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:52.916266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.191566+0000","last_change":"2024-03-04T16:07:36.224345+0000","last_active":"2024-03-04T16:09:40.191566+0000","last_peered":"2024-03-04T16:09:40.191566+0000","last_clean":"2024-03-04T16:09:40.191566+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:09:40.191566+0000","last_undegraded":"2024-03-04T16:09:40.191566+0000","last_fullsized":"2024-03-04T16:09:40.191566+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:06:51.144836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:06:51.144836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:09:41.713916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-04T16:09:40.183696+0000","last_change":"2024-03-04T16:09:08.714773+0000","last_active":"2024-03-04T16:09:40.183696+0000","last_peered":"2024-03-04T16:09:40.183696+0000","last_clean":"2024-03-04T16:09:40.183696+0000","last_became_active":"2024-03-04T16:02:45.743793+0000","last_became_peered":"2024-03-04T16:02:45.743793+0000","last_unstale":"2024-03-04T16:09:40.183696+0000","last_undegraded":"2024-03-04T16:09:40.183696+0000","last_fullsized":"2024-03-04T16:09:40.183696+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:26.690427+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4323082240,"num_objects":65965,"num_object_clones":0,"num_object_copies":197895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65965,"num_write_kb":4221760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12953583616,"data_stored":12953583616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65965,"ondisk_log_size":65965,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":412,"num_read_kb":407,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":440,"ondisk_log_size":440,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070552,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1481524,"kb_used_data":1297640,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92263628,"statfs":{"total":95995035648,"available":94477955072,"internally_reserved":0,"allocated":1328783360,"data_stored":1328323180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364892,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1440988,"kb_used_data":1293200,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92304164,"statfs":{"total":95995035648,"available":94519463936,"internally_reserved":0,"allocated":1324236800,"data_stored":1323798884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2030668,"kb_used_data":1814848,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91714484,"statfs":{"total":95995035648,"available":93915631616,"internally_reserved":0,"allocated":1858404352,"data_stored":1857918292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738490,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1805168,"kb_used_data":1571304,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":91939984,"statfs":{"total":95995035648,"available":94146543616,"internally_reserved":0,"allocated":1609015296,"data_stored":1608554460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730268,"kb_used_data":1550544,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92014884,"statfs":{"total":95995035648,"available":94223241216,"internally_reserved":0,"allocated":1587757056,"data_stored":1587319316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1876924,"kb_used_data":1597236,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91868228,"statfs":{"total":95995035648,"available":94073065472,"internally_reserved":0,"allocated":1635569664,"data_stored":1635096106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":286390728},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953574,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1453108,"kb_used_data":1313640,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92292044,"statfs":{"total":95995035648,"available":94507053056,"internally_reserved":0,"allocated":1345167360,"data_stored":1344705314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626530,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2253528,"kb_used_data":2077968,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91491624,"statfs":{"total":95995035648,"available":93687422976,"internally_reserved":0,"allocated":2127839232,"data_stored":2127401098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1627783168,"data_stored":1627783168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1890058240,"data_stored":1890058240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1613430784,"data_stored":1613430784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1637089280,"data_stored":1637089280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1360199680,"data_stored":1360199680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2136408064,"data_stored":2136408064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1336606720,"data_stored":1336606720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1352007680,"data_stored":1352007680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:09:48.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-04T16:09:48.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:48.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:48.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:48.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:48.940 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:48.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:49.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:49.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 41 16 73858 73842 112.549 77.875 0.00413625 0.0088802 2024-03-04T16:09:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:49.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:49.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.227 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:49.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:49.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:49.358 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:49.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:49.448 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:49.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:49.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.538 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:49.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:49.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:49.672 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:49.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:49.804 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:49.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:49.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:49.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:50.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:50.066 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:50.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 42 16 75061 75045 111.66 75.1875 0.0057339 0.00894772 2024-03-04T16:09:50.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:50.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:50.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:50.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:50.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:50.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:50.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:50.623 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:50.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:50.771 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:50.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:50.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:50.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:51.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:51.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:51.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 43 16 76292 76276 110.852 76.9375 0.00685173 0.00901507 2024-03-04T16:09:51.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:51.192 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:51.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:51.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:51 smithi038 ceph-mon[29779]: pgmap v298: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-04T16:09:51.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:51 smithi038 ceph-mon[29779]: osdmap e182: 8 total, 8 up, 8 in 2024-03-04T16:09:51.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:51 smithi038 ceph-mon[38113]: pgmap v298: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-04T16:09:51.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:51 smithi038 ceph-mon[38113]: osdmap e182: 8 total, 8 up, 8 in 2024-03-04T16:09:51.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:51.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:51.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:51.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:09:51.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:51 smithi176 ceph-mon[33619]: pgmap v298: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-04T16:09:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:51 smithi176 ceph-mon[33619]: osdmap e182: 8 total, 8 up, 8 in 2024-03-04T16:09:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.662 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:51.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:51.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:51.852 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:51.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:51.988 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:51.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:52.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 44 16 77537 77521 110.101 77.8125 0.00462941 0.00907738 2024-03-04T16:09:52.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:52.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:52.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:52.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:52.314 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:52.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:52.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:52.422 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:52.524 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:52.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:09:52.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:52.691 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:52.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:52.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:52.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:52.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:52.969 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:52.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:52.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:53.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:53.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 45 16 78798 78782 109.406 78.8125 0.0168799 0.00913552 2024-03-04T16:09:53.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:53.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:53.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:09:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:09:53.206 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.240 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:53.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:53.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:53 smithi038 ceph-mon[29779]: pgmap v300: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 819 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-04T16:09:53.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:53 smithi038 ceph-mon[38113]: pgmap v300: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 819 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-04T16:09:53.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:53.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:53.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:53.405 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:09:53.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:53.539 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:53 smithi176 ceph-mon[33619]: pgmap v300: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 819 B/s rd, 61 MiB/s wr, 984 op/s 2024-03-04T16:09:53.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:53.655 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:09:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:09:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:09:53.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:53.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:53.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:53.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:53.957 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:53.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:54.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:54.072 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:09:54.073 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:09:54.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 46 16 79929 79913 108.564 70.6875 0.0261746 0.00920615 2024-03-04T16:09:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:54.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:54.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:54.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:54.227 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:09:54.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:54.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3077884661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[29779]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:54.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3077884661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:54 smithi038 ceph-mon[38113]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.364 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:54.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:54.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:54.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:09:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3077884661' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:54 smithi176 ceph-mon[33619]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:09:54.609 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:54.640 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:54.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:54.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:54.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:54.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:54.924 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:54.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:55.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.065 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:55.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:55.079 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 47 16 81203 81187 107.948 79.625 0.00992594 0.00925865 2024-03-04T16:09:55.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:55.193 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:55.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:55.333 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:55.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:55.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:55.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[29779]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.54k op/s 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[29779]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[29779]: osdmap e183: 8 total, 8 up, 8 in 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[29779]: osdmap e184: 8 total, 8 up, 8 in 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[38113]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.54k op/s 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[38113]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:09:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[38113]: osdmap e183: 8 total, 8 up, 8 in 2024-03-04T16:09:55.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:55 smithi038 ceph-mon[38113]: osdmap e184: 8 total, 8 up, 8 in 2024-03-04T16:09:55.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:55 smithi176 ceph-mon[33619]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.54k op/s 2024-03-04T16:09:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:55 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:09:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:55 smithi176 ceph-mon[33619]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:09:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:55 smithi176 ceph-mon[33619]: osdmap e183: 8 total, 8 up, 8 in 2024-03-04T16:09:55.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:55 smithi176 ceph-mon[33619]: osdmap e184: 8 total, 8 up, 8 in 2024-03-04T16:09:55.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:55.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:55.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:55.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:09:55.848 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:55.930 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:55.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:56.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:56.070 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:56.080 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 48 16 82377 82361 107.228 73.375 0.0155791 0.00932041 2024-03-04T16:09:56.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:56.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:56.227 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:09:56.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:56.365 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:56.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:56.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:56 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:09:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:56 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:09:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:56 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:09:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:56 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:09:56.498 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:56 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:09:56.498 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:56 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:09:56.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:56.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.651 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:56.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:56.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:56.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:56.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:56.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:56.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:57.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.075 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:57.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:57.080 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 49 16 83570 83554 106.561 74.5625 0.0137337 0.00937894 2024-03-04T16:09:57.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.216 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:57.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:57.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:57.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:57.492 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: pgmap v304: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:09:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:09:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:09:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:09:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[29779]: 8.4 scrub starts 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: pgmap v304: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:09:57.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:57 smithi038 ceph-mon[38113]: 8.4 scrub starts 2024-03-04T16:09:57.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: pgmap v304: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:09:57.521 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:57 smithi176 ceph-mon[33619]: 8.4 scrub starts 2024-03-04T16:09:57.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:57.639 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:57.741 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.785 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:57.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:57.886 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:57.938 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:57.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:58.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:58.080 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 50 16 84802 84786 105.969 77 0.0139093 0.00943151 2024-03-04T16:09:58.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:58.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:58.215 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:09:58.215 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:09:58.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:58.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:09:58.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:09:58.228 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:09:58.229 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:09:58.229 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 8.4 scrub ok 2024-03-04T16:09:58.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:09:58.370 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:09:58.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 8.4 scrub ok 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: pgmap v305: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 706 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-04T16:09:58.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:09:58.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:09:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: pgmap v305: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 706 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-04T16:09:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:09:58.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:58 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:09:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:58.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:58.558 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 8.4 scrub ok 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: pgmap v305: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 706 B/s rd, 57 MiB/s wr, 910 op/s 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:09:58.559 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:58 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:09:58.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:58.644 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:58.746 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:58.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:58.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:09:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:58.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:09:58.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:09:59.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:09:59.051 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:09:59.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 51 16 85775 85759 105.084 60.8125 0.0110475 0.0095107 2024-03-04T16:09:59.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:09:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.200 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:59.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:09:59.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:09:59.333 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:09:59.340 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:09:59.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:09:59.456 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:09:59 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:09:59.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:09:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:09:59.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:09:59.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:09:59 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:09:59.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:09:59.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:09:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.635 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:59.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:09:59.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:09:59.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:09:59.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:09:59.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:09:59.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:09:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:09:59.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:00.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:00.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 52 16 86830 86814 104.33 65.9375 0.019057 0.00958008 2024-03-04T16:10:00.087 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:00.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:00.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:00.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:00.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:10:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: pgmap v306: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-04T16:10:00.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:00 smithi176 ceph-mon[33619]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-04T16:10:00.372 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:00.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:00.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:10:00.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: pgmap v306: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[29779]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:10:00.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:10:00.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: pgmap v306: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:10:00.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:10:00.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:10:00.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-04T16:10:00.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:00 smithi038 ceph-mon[38113]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-04T16:10:00.511 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:00.608 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:10:00.613 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:00.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:00.682 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:10:00.708 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:10:00.742 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:10:00.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:00.788 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:10:00.790 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:00.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:10:00.881 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:10:00.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:00.918 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:10:00.920 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:10:00.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:00.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:01.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:01.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:01.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 53 16 87792 87776 103.495 60.125 0.0213053 0.00965728 2024-03-04T16:10:01.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:01.217 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:01.232 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:10:01.232 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:10:01.232 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:10:01.233 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:10:01.233 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:10:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 1 pgp_num to 1 2024-03-04T16:10:01.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:01.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3080296318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2995361629' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:01 smithi176 ceph-mon[33619]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:01.494 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:01.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:01.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:01.632 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:01.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3080296318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2995361629' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[29779]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:10:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:10:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:10:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:10:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3080296318' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2995361629' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:01 smithi038 ceph-mon[38113]: from='client.35090 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:10:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.767 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:01.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:01.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:01.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:01.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:01.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:10:02.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:02.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:02.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 54 16 88979 88963 102.953 74.1875 0.0121974 0.00970796 2024-03-04T16:10:02.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:02.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:02.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:02.337 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:02.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.471 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:02.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:02.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.607 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:02.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:10:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: osdmap e185: 8 total, 8 up, 8 in 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: pgmap v308: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 52 MiB/s wr, 835 op/s 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:10:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:10:02.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:10:02.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:10:02.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:10:02.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:10:02.663 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: osdmap e185: 8 total, 8 up, 8 in 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: pgmap v308: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 52 MiB/s wr, 835 op/s 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:10:02.664 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: from='client.35090 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: osdmap e185: 8 total, 8 up, 8 in 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: pgmap v308: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 707 B/s rd, 52 MiB/s wr, 835 op/s 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:10:02.665 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:02 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:10:02.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:02.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:02.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:02.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:02.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:02.991 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:03.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:03.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 55 15 90186 90171 102.453 75.5 0.0162736 0.00975562 2024-03-04T16:10:03.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:03.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:03.163 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:03.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:03.299 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:03.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:10:03.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:03.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:10:03.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:10:03.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:10:03.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:10:03.421 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:03.448 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:03.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:03.563 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:03 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:10:03.563 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:03 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:10:03.563 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:03 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:10:03.563 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:03 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:10:03.563 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:03.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:03.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:03.604 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:03.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:03.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:03.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:03.869 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:03.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:03.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:03.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:04.081 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 56 15 91332 91317 101.903 71.625 0.0105512 0.00980814 2024-03-04T16:10:04.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:04.139 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:04.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.272 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:8980 7.1 7:85485f01:::benchmark_data_smithi038_77863_object8979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 313.26458341699998, 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.075430288999999998, 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 8980 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004852+0000", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004853+0000", 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1570000000000001e-06 2024-03-04T16:10:04.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004884+0000", 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0892999999999997e-05 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004896+0000", 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1905999999999999e-05 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004903+0000", 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4950000000000002e-06 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.004952+0000", 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8758999999999999e-05 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005044+0000", 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2423999999999995e-05 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005176+0000", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131514 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.005795+0000", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00061942300000000002 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.006112+0000", 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00031637899999999999 2024-03-04T16:10:04.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080171+0000", 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.074059199000000006 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080236+0000", 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5553000000000001e-05 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:51.080282+0000", 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5587000000000002e-05 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23564 7.1 7:8f2d6c93:::benchmark_data_smithi038_77863_object23563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 307.05217028700002, 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067845241000000001, 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.279 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23564 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217265+0000", 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217273+0000", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7649999999999993e-06 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217320+0000", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7490000000000001e-05 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217331+0000", 2024-03-04T16:10:04.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0560000000000001e-05 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217337+0000", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8399999999999997e-06 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217396+0000", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8226000000000001e-05 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217522+0000", 2024-03-04T16:10:04.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012583599999999999 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.217665+0000", 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014378200000000001 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.219333+0000", 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001667647 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.219511+0000", 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017759899999999999 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285008+0000", 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065497256000000004 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285064+0000", 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6240999999999999e-05 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.284 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285110+0000", 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5998999999999998e-05 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:23579 7.1 7:8c29c604:::benchmark_data_smithi038_77863_object23578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 307.047388023, 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063222441000000004, 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.285 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 23579 2024-03-04T16:10:04.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222047+0000", 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222049+0000", 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.437e-06 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222105+0000", 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6329000000000002e-05 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222116+0000", 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1518000000000001e-05 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222122+0000", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0340000000000002e-06 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222173+0000", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0720999999999998e-05 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222270+0000", 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6972999999999995e-05 2024-03-04T16:10:04.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.222399+0000", 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128999 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.223163+0000", 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076355399999999995 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.223487+0000", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000324729 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285180+0000", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.061692898000000003 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285240+0000", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9389999999999999e-05 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.285270+0000", 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9859e-05 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24064 7.1 7:8a6e839d:::benchmark_data_smithi038_77863_object24063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 306.77195917199998, 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12912359700000001, 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.292 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24064 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497478+0000", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.956e-06 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497544+0000", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6097000000000001e-05 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497559+0000", 2024-03-04T16:10:04.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4935e-05 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497568+0000", 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9970000000000004e-06 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497637+0000", 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8853e-05 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497782+0000", 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014473599999999999 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497906+0000", 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124091 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499051+0000", 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011451409999999999 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499189+0000", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013806699999999999 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626465+0000", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12727575699999999 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626544+0000", 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.907e-05 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626600+0000", 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5896999999999997e-05 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24066 7.1 7:873d823e:::benchmark_data_smithi038_77863_object24065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 306.77028959, 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.130573199, 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.298 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24066 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499147+0000", 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.274e-06 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499199+0000", 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2524000000000001e-05 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499208+0000", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6740000000000003e-06 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499214+0000", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5999999999999997e-06 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499248+0000", 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4187999999999999e-05 2024-03-04T16:10:04.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499344+0000", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6173000000000002e-05 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499488+0000", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014452200000000001 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.500600+0000", 2024-03-04T16:10:04.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011114740000000001 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.501582+0000", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00098159099999999997 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629612+0000", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12803046100000001 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629673+0000", 2024-03-04T16:10:04.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1291000000000007e-05 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629719+0000", 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5426999999999997e-05 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24076 7.1 7:802e523e:::benchmark_data_smithi038_77863_object24075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:10:04.304 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 306.76551689299998, 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12599276100000001, 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24076 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503920+0000", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4190000000000001e-06 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504014+0000", 2024-03-04T16:10:04.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4440999999999999e-05 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504025+0000", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0706999999999999e-05 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504032+0000", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.002e-06 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504086+0000", 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4079999999999997e-05 2024-03-04T16:10:04.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504200+0000", 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114207 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504319+0000", 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011859899999999999 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505150+0000", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083150499999999996 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505431+0000", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028036800000000002 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629771+0000", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12434076600000001 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629841+0000", 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9821999999999997e-05 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629911+0000", 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9845000000000006e-05 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24080 7.1 7:85c22b13:::benchmark_data_smithi038_77863_object24079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:04.310 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 306.75873840700001, 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11934697599999999, 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24080 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510699+0000", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0370000000000001e-06 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510740+0000", 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1344000000000003e-05 2024-03-04T16:10:04.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510753+0000", 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3010000000000001e-05 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510760+0000", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3000000000000004e-06 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510820+0000", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9786e-05 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510959+0000", 2024-03-04T16:10:04.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013839899999999999 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511076+0000", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117656 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511948+0000", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00087144999999999996 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.512095+0000", 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147682 2024-03-04T16:10:04.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629957+0000", 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11786121500000001 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630007+0000", 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0105000000000002e-05 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630044+0000", 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6992000000000001e-05 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38263 7.1 7:894d19c1:::benchmark_data_smithi038_77863_object38262:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 299.09595031800001, 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067151086999999998, 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.317 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38263 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173485+0000", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173487+0000", 2024-03-04T16:10:04.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8250000000000001e-06 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173547+0000", 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9956e-05 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173558+0000", 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1263e-05 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173564+0000", 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9290000000000003e-06 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173616+0000", 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2658999999999998e-05 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173723+0000", 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000106154 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.173821+0000", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8696000000000005e-05 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.174908+0000", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010863069999999999 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.175260+0000", 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035221599999999998 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240529+0000", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.065269674999999999 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240588+0000", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8378000000000002e-05 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.240636+0000", 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8028999999999997e-05 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.322 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:38267 7.1 7:8d7ce7cb:::benchmark_data_smithi038_77863_object38266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 299.09116497399998, 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.064004921000000006, 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 38267 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.323 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178270+0000", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178273+0000", 2024-03-04T16:10:04.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1499999999999999e-06 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178332+0000", 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8739999999999997e-05 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178365+0000", 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3142999999999998e-05 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178380+0000", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5199e-05 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178504+0000", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012342000000000001 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178717+0000", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000213258 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.178801+0000", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3704999999999999e-05 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.179805+0000", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010037799999999999 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.179905+0000", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000100589 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242182+0000", 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062276435999999998 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242237+0000", 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4957000000000003e-05 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:05.242275+0000", 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8544000000000002e-05 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.328 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83920 7.1 7:8d8feee9:::benchmark_data_smithi038_77863_object83919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 263.61621660499998, 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10767995700000001, 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83920 2024-03-04T16:10:04.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653220+0000", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.545e-06 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653270+0000", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9568999999999999e-05 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653281+0000", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1192999999999999e-05 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653296+0000", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4987999999999999e-05 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653345+0000", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8775e-05 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653480+0000", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013508899999999999 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653647+0000", 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167004 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656243+0000", 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025961299999999999 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656442+0000", 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199464 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760825+0000", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.104383084 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760863+0000", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7704999999999999e-05 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760898+0000", 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5410999999999997e-05 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83925 7.1 7:87a9c3a2:::benchmark_data_smithi038_77863_object83924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 263.612865204, 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.107333128, 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83925 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656572+0000", 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.778e-06 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656615+0000", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3161000000000002e-05 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656626+0000", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1065e-05 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656632+0000", 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7389999999999996e-06 2024-03-04T16:10:04.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656674+0000", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656764+0000", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0019999999999995e-05 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656883+0000", 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119466 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659071+0000", 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021878219999999999 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659140+0000", 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8551000000000001e-05 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763805+0000", 2024-03-04T16:10:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.341 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:04.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10466587199999999 2024-03-04T16:10:04.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763859+0000", 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3298999999999999e-05 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763903+0000", 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4301999999999997e-05 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84327 7.1 7:8c4037d9:::benchmark_data_smithi038_77863_object84326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 263.07501218700003, 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15446056899999999, 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84327 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194425+0000", 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-03-04T16:10:04.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194465+0000", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9663000000000003e-05 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194479+0000", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4025999999999999e-05 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194488+0000", 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8810000000000002e-06 2024-03-04T16:10:04.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194551+0000", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2869999999999999e-05 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194677+0000", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012619000000000001 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194823+0000", 2024-03-04T16:10:04.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145477 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197906+0000", 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030836819999999999 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197926+0000", 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355999999999999e-05 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348763+0000", 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15083737699999999 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348834+0000", 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1464999999999994e-05 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348883+0000", 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9094000000000001e-05 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.348 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84339 7.1 7:81128b26:::benchmark_data_smithi038_77863_object84338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 263.053417986, 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132991251, 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84339 2024-03-04T16:10:04.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216020+0000", 2024-03-04T16:10:04.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6180000000000002e-06 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216053+0000", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3401000000000001e-05 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216067+0000", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3816e-05 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216075+0000", 2024-03-04T16:10:04.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6839999999999996e-06 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217281+0000", 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001206181 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217443+0000", 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162063 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.352 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217620+0000", 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017687900000000001 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219824+0000", 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022046380000000001 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219965+0000", 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014102400000000001 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.353 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348936+0000", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128970469 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348971+0000", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5068999999999999e-05 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.349008+0000", 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7409e-05 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.354 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:103907 7.1 7:819e1ab5:::benchmark_data_smithi038_77863_object103906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 241.36350656299999, 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.066058037999999999, 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 103907 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.355 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905929+0000", 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.356 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.905931+0000", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0420000000000001e-06 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906024+0000", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3833999999999996e-05 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906037+0000", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2571e-05 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.357 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906045+0000", 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7449999999999995e-06 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906078+0000", 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3556999999999997e-05 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906187+0000", 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108799 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.358 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.906335+0000", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014748300000000001 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.908898+0000", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002563358 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.908915+0000", 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7507999999999999e-05 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.359 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971911+0000", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.062995491000000001 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971954+0000", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3084999999999998e-05 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:02.971987+0000", 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2564999999999997e-05 2024-03-04T16:10:04.360 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163755 7.4 7:20abbf33:::benchmark_data_smithi038_77863_object45487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 179.05768763699999, 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1434231819999998, 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163755 2024-03-04T16:10:04.361 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.362 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211749+0000", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4980000000000001e-06 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211755+0000", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9410000000000004e-06 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211767+0000", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1882000000000001e-05 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.363 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211775+0000", 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9910000000000002e-06 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211836+0000", 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1597e-05 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211846+0000", 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6569999999999992e-06 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.364 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211462+0000", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99961569800000005 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211477+0000", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4766999999999999e-05 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.337886+0000", 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1264093390000001 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.365 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339470+0000", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015841239999999999 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339721+0000", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025078399999999999 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351914+0000", 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012193641 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.366 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351962+0000", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8062000000000002e-05 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352023+0000", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0387999999999998e-05 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355003+0000", 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029802489999999999 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.367 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355055+0000", 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2178000000000001e-05 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355171+0000", 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115386 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.368 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163761 7.4 7:2ccd0a78:::benchmark_data_smithi038_77863_object45493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 179.04184997499999, 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1332310689999998, 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163761 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.369 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227587+0000", 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.812e-06 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.370 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227592+0000", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227603+0000", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0251000000000001e-05 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227610+0000", 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9650000000000002e-06 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.371 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227662+0000", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2834999999999998e-05 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227672+0000", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8989999999999996e-06 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211510+0000", 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98383736899999996 2024-03-04T16:10:04.372 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211518+0000", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5059999999999995e-06 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352101+0000", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.140582808 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352222+0000", 2024-03-04T16:10:04.373 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120686 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352361+0000", 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139345 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355375+0000", 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030134020000000001 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.374 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.356429+0000", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001054234 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.358238+0000", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018089410000000001 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360641+0000", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024032419999999999 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.375 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360703+0000", 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2532000000000004e-05 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360816+0000", 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011275100000000001 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:177490 7.1 7:82ecc071:::benchmark_data_smithi038_77863_object59222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:10:04.376 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 161.805627993, 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10186015399999999, 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177490 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.377 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463809+0000", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.767e-06 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463814+0000", 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:10:04.378 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463824+0000", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6409999999999997e-06 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463831+0000", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1049999999999997e-06 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465019+0000", 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011877140000000001 2024-03-04T16:10:04.379 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465136+0000", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011716400000000001 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465249+0000", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011281399999999999 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.468002+0000", 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027534370000000001 2024-03-04T16:10:04.380 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.469006+0000", 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010038709999999999 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.381 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565552+0000", 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.096545858999999998 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565616+0000", 2024-03-04T16:10:04.382 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4599999999999998e-05 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565667+0000", 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0862999999999997e-05 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:179503 7.1 7:86d99f9c:::benchmark_data_smithi038_77863_object61235:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:25.506567+0000", 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 158.76286813199999, 2024-03-04T16:10:04.383 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.063571014999999995, 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179503 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506567+0000", 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.384 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506567+0000", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506569+0000", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.086e-06 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506576+0000", 2024-03-04T16:10:04.385 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8349999999999998e-06 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506592+0000", 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6144999999999999e-05 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506602+0000", 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5589999999999994e-06 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.386 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506684+0000", 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2139999999999996e-05 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506822+0000", 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013784999999999999 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.506941+0000", 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119448 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.387 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.509550+0000", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026091209999999998 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.509722+0000", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017159599999999999 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.570002+0000", 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.060280067999999999 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.388 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.570091+0000", 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8900999999999995e-05 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:25.570138+0000", 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7265999999999998e-05 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.389 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:222378 7.1 7:83b38929:::benchmark_data_smithi038_77863_object104110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 110.170442721, 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090076495000000006, 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 222378 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.390 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098995+0000", 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6989999999999999e-06 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.391 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099001+0000", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9739999999999999e-06 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099026+0000", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4769999999999998e-05 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099034+0000", 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0760000000000003e-06 2024-03-04T16:10:04.392 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099091+0000", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6709000000000002e-05 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099742+0000", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00065097099999999997 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099852+0000", 2024-03-04T16:10:04.393 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110408 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102653+0000", 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028013769999999999 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102881+0000", 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227686 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.394 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.188963+0000", 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.086082045999999995 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189019+0000", 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6125000000000002e-05 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189069+0000", 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9654000000000003e-05 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.395 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:56999 8.2 8:401e7e3c:::benchmark_data_smithi038_107057_object56998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:09:36.248364+0000", 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 28.021071509999999, 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.070685493000000002, 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 56999 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.396 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248364+0000", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248364+0000", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248365+0000", 2024-03-04T16:10:04.397 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.849e-06 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248416+0000", 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0226999999999999e-05 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248430+0000", 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4470999999999999e-05 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:04.398 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248438+0000", 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6930000000000007e-06 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248490+0000", 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1903000000000001e-05 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248626+0000", 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013607499999999999 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.399 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.248758+0000", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000132526 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.250577+0000", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018183749999999999 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.250952+0000", 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037525200000000002 2024-03-04T16:10:04.400 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.318935+0000", 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.067983392000000004 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.318998+0000", 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2918999999999998e-05 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:09:36.319049+0000", 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0810999999999999e-05 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:04.403 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:10:04.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:04.476 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:04.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:04.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:04 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:10:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:04 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:10:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:04 smithi176 ceph-mon[33619]: pgmap v309: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:04.611 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:04.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[29779]: pgmap v309: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:10:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:04 smithi038 ceph-mon[38113]: pgmap v309: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:04.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:04.848 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:04.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:04.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:04.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.051 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:05.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:05.085 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 57 16 92550 92534 101.442 76.0625 0.0148488 0.00985296 2024-03-04T16:10:05.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:10:05.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:05.192 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:10:05.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:05.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:05.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:05.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:05.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:05.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:10:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[29779]: osdmap e186: 8 total, 8 up, 8 in 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:10:05.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:10:05.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:05 smithi038 ceph-mon[38113]: osdmap e186: 8 total, 8 up, 8 in 2024-03-04T16:10:05.547 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:05.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:10:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:10:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:10:05.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:10:05.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:10:05.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:10:05.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:10:05.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:10:05.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:10:05.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:10:05.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:10:05.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:10:05.611 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:05 smithi176 ceph-mon[33619]: osdmap e186: 8 total, 8 up, 8 in 2024-03-04T16:10:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.634 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:05.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:05.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:05.775 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:05.839 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:05.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:05.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:05.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:06.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:06.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:06.085 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 58 16 93572 93556 100.794 63.875 0.0202754 0.00991609 2024-03-04T16:10:06.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:06.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:06.341 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:10:06.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:06.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:06.504 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3223654423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[29779]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3223654423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:06 smithi038 ceph-mon[38113]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:10:06.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:10:06.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3223654423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:06 smithi176 ceph-mon[33619]: from='client.35102 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.645 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:06.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:06.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:06.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:06.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:06.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:06.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:06.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:07.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:07.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:07.085 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 59 16 94651 94635 100.229 67.4375 0.0174404 0.00997159 2024-03-04T16:10:07.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:07.199 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:07.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:07.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:07.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.468 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:07.569 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:07.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: osdmap e187: 8 total, 8 up, 8 in 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:07.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: osdmap e187: 8 total, 8 up, 8 in 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:07.739 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:10:07.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:10:07.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: from='client.35102 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:07.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: osdmap e187: 8 total, 8 up, 8 in 2024-03-04T16:10:07.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:07.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:07.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:07.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:07.899 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:07.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:08.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:08.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:08.085 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:10:08.086456+0000 min lat: 0.00119552 max lat: 0.122782 avg lat: 0.0100135 2024-03-04T16:10:08.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:10:08.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 60 16 95857 95841 99.8147 75.375 0.0201234 0.0100135 2024-03-04T16:10:08.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.184 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:08.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:08.285 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:08.324 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:08.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:08.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:08.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:08.553 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:10:08.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:08.589 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: pgmap v313: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 830 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:10:08.590 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:10:08.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:10:08.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: pgmap v313: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 830 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:10:08.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:10:08.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:08 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:10:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: pgmap v313: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 830 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:10:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:08 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:10:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:08.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:08.628 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:10:08.642 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:10:08.643 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:10:08.643 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:10:08.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd out 2 2024-03-04T16:10:08.669 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:10:08.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:08.717 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:10:08.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:08.767 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:10:08.841 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:10:08.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:08.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:10:08.891 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:10:08.891 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:10:08.891 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:10:08.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:10:08.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:10:08.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:10:08.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:10:08.892 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:10:08.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:08.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:09.008 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:09.009 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:09.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:09.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:09.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 61 15 96928 96913 99.2767 67 0.00514056 0.0100683 2024-03-04T16:10:09.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:09.180 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:09.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:09.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:09.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:10:09.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:09.462 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:09.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:09.608 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 6.15 deep-scrub starts 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 6.15 deep-scrub ok 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 5.1f deep-scrub starts 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 5.1f deep-scrub ok 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3352861598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:10:09.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:09 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:10:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:09.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 6.15 deep-scrub starts 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 6.15 deep-scrub ok 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 5.1f deep-scrub starts 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 5.1f deep-scrub ok 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3352861598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 6.15 deep-scrub starts 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 6.15 deep-scrub ok 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 5.1f deep-scrub starts 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 5.1f deep-scrub ok 2024-03-04T16:10:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3352861598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:10:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:09 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:10:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:09.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:09.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:09.862 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:09.911 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:09.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:10.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:10.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:10.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 62 16 98045 98029 98.7999 69.75 0.00701165 0.0101164 2024-03-04T16:10:10.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:10.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:10.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:10:10.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:10.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:10.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:10.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.495 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:10.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:10.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:10.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:10 smithi176 ceph-mon[33619]: pgmap v314: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:10.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:10 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:10:10.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:10 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:10:10.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:10 smithi176 ceph-mon[33619]: osdmap e188: 8 total, 8 up, 8 in 2024-03-04T16:10:10.620 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.634 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:10.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:10.657 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:10.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[29779]: pgmap v314: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:10:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:10:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[29779]: osdmap e188: 8 total, 8 up, 8 in 2024-03-04T16:10:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[38113]: pgmap v314: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-04T16:10:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:10:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:10:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:10 smithi038 ceph-mon[38113]: osdmap e188: 8 total, 8 up, 8 in 2024-03-04T16:10:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.784 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:10.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:10.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:10.915 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:10.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:11.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:11.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:11.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 63 16 99201 99185 98.3784 72.25 0.0161553 0.0101591 2024-03-04T16:10:11.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:11.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:11.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:11.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.324 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:11.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:11.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:11.464 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:11.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:11.607 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:11.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:10:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:10:11.748 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:11.823 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:11 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:10:11.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:11 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:10:11.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:10:11.824 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:10:11.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:10:11.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:11.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:11.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:11.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:12.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:12.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 64 16 100400 100384 98.0121 74.9375 0.0110683 0.0101974 2024-03-04T16:10:12.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:12.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:12.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:12.308 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:12.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:12.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:12.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[29779]: pgmap v316: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 819 B/s rd, 57 MiB/s wr, 908 op/s 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:10:12.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:10:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:10:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[38113]: pgmap v316: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 819 B/s rd, 57 MiB/s wr, 908 op/s 2024-03-04T16:10:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:10:12.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:12 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:10:12.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:12.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:10:12.583 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:10:12.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-mon[33619]: pgmap v316: 148 pgs: 148 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 819 B/s rd, 57 MiB/s wr, 908 op/s 2024-03-04T16:10:12.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:10:12.584 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:10:12.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:12.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.730 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:12.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:12.776 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:12.777 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:12.831 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:12.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:12.867 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:12.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:12.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:13.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:13.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 65 16 101560 101544 97.6196 72.5 0.0095134 0.0102388 2024-03-04T16:10:13.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:13.143 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:13.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.283 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:13.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:13.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:13.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:13.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:13.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:10:13.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:13.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:10:13.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:13.545 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:10:13.546 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:13 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:10:13.567 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:13.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:13.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.729 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:13.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:13.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:13.850 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:13 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:10:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:13 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:10:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:13 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:10:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:13 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:10:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:13.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:13.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:14.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:14.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:14.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 66 16 102638 102622 97.1613 67.375 0.00798518 0.0102871 2024-03-04T16:10:14.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:14.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:14.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:14.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:14.333 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:14.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.481 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:14.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:14.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:14 smithi176 ceph-mon[33619]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-04T16:10:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:14 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:10:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:14 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:10:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:14.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[29779]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[38113]: pgmap v317: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.43k op/s 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:10:14.722 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:14 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:10:14.725 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:14.729 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:14.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:14.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:14.899 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:14.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:15.000 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:15.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:15.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 67 16 103527 103511 96.5404 55.5625 0.00807745 0.010353 2024-03-04T16:10:15.140 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:15.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:15.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:15.310 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:15.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:15.439 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:15.454 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:10:15.541 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:15.573 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:15.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:10:15.680 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:10:15.706 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:15.751 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:15 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:10:15.751 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:15 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:10:15.751 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:10:15.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.845 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:15.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:15.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:15 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:10:15.897+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e189 definitely_dead 0 2024-03-04T16:10:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:15.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:15.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:16.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:16.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 68 16 104518 104502 96.0315 61.9375 0.0167011 0.010408 2024-03-04T16:10:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:16.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:16.281 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:16.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:16.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:16.445 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.2. 2024-03-04T16:10:16.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:16.555 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:16.555 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:16.569 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:16.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:16.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: osdmap e189: 8 total, 8 up, 8 in 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[29779]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:10:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:10:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: osdmap e189: 8 total, 8 up, 8 in 2024-03-04T16:10:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-04T16:10:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:16 smithi038 ceph-mon[38113]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-04T16:10:16.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:16.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1806070282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:10:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:10:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: osdmap e189: 8 total, 8 up, 8 in 2024-03-04T16:10:16.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-03-04T16:10:16.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:16 smithi176 ceph-mon[33619]: Client client.admin marked osd.2 out, while it was still marked up 2024-03-04T16:10:16.922 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:16.945 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:16.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:17.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:17.071 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:17.086 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 69 16 104962 104946 95.0419 27.75 0.00324338 0.0104243 2024-03-04T16:10:17.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:17.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:17.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:17.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:10:17.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:17.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:17.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:17.501 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:17.539 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:17.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.627 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:17.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:17.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-04T16:10:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:17 smithi038 ceph-mon[29779]: osdmap e190: 8 total, 8 up, 7 in 2024-03-04T16:10:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-04T16:10:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:17 smithi038 ceph-mon[38113]: osdmap e190: 8 total, 8 up, 7 in 2024-03-04T16:10:17.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:17.782 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/275815996' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-03-04T16:10:17.782 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:17 smithi176 ceph-mon[33619]: osdmap e190: 8 total, 8 up, 7 in 2024-03-04T16:10:17.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:17.878 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:17.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:17.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:18.002 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:18.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 70 16 104962 104946 93.6843 0 - 0.0104243 2024-03-04T16:10:18.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:18.126 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:18.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:18.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:18.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:18.382 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:18.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:18.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:18 smithi176 ceph-mon[33619]: pgmap v321: 148 pgs: 148 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 706 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-04T16:10:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:18 smithi176 ceph-mon[33619]: osdmap e191: 8 total, 8 up, 7 in 2024-03-04T16:10:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:18 smithi176 ceph-mon[33619]: osdmap e192: 8 total, 8 up, 7 in 2024-03-04T16:10:18.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:18.640 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:18.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:18.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[29779]: pgmap v321: 148 pgs: 148 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 706 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-04T16:10:18.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[29779]: osdmap e191: 8 total, 8 up, 7 in 2024-03-04T16:10:18.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[29779]: osdmap e192: 8 total, 8 up, 7 in 2024-03-04T16:10:18.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[38113]: pgmap v321: 148 pgs: 148 active+clean; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 706 B/s rd, 50 MiB/s wr, 799 op/s 2024-03-04T16:10:18.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[38113]: osdmap e191: 8 total, 8 up, 7 in 2024-03-04T16:10:18.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:18 smithi038 ceph-mon[38113]: osdmap e192: 8 total, 8 up, 7 in 2024-03-04T16:10:18.776 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:18.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:18.916 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:18.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:19.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:19.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 71 16 105330 105314 92.6888 11.5 0.0233022 0.0107831 2024-03-04T16:10:19.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:19.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.243 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:19.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:19.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:19.348 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:19.348 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:19.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:19.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:19.531 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 189'6483 2024-03-04T16:10:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 189'6648 2024-03-04T16:10:19.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 189'6569 2024-03-04T16:10:19.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 189'6592 2024-03-04T16:10:19.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 189'6544 2024-03-04T16:10:19.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:19 smithi176 ceph-mon[33619]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 189'6479 2024-03-04T16:10:19.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:19.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:19.708 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 189'6483 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 189'6648 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 189'6569 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 189'6592 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 189'6544 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[29779]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 189'6479 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.4 starting backfill to osd.6 from (0'0,0'0] MAX to 189'6483 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.1 starting backfill to osd.5 from (0'0,0'0] MAX to 189'6648 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.d starting backfill to osd.6 from (0'0,0'0] MAX to 189'6569 2024-03-04T16:10:19.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.e starting backfill to osd.7 from (0'0,0'0] MAX to 189'6592 2024-03-04T16:10:19.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.a starting backfill to osd.3 from (0'0,0'0] MAX to 189'6544 2024-03-04T16:10:19.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:19 smithi038 ceph-mon[38113]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 189'6479 2024-03-04T16:10:19.751 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:10:19.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:19.793 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:10:19.801 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:19.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:10:19.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:10:19.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:19.931 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:10:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:19.944 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:19.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:19.979 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:10:20.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:10:20.021 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:10:20.027 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:10:20.028 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:10:20.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:20.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 72 15 106385 106370 92.3181 66 0.00788314 0.0108272 2024-03-04T16:10:20.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.102 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:20.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:20.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.250 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:20.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:20.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:20.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:20.455 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:20.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:20.553 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:20.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:20.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:20.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:20.785 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:20 smithi176 ceph-mon[33619]: pgmap v324: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 940 op/s; 18/315444 objects degraded (0.006%); 171 B/s, 9 objects/s recovering 2024-03-04T16:10:20.785 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:20 smithi176 ceph-mon[33619]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:10:20.785 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:20 smithi176 ceph-mon[33619]: Health check failed: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:20.785 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2671755020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:20.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.824 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:20.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[29779]: pgmap v324: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 940 op/s; 18/315444 objects degraded (0.006%); 171 B/s, 9 objects/s recovering 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[29779]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[29779]: Health check failed: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2671755020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[38113]: pgmap v324: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 940 op/s; 18/315444 objects degraded (0.006%); 171 B/s, 9 objects/s recovering 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[38113]: Health check failed: Reduced data availability: 3 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[38113]: Health check failed: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded (PG_DEGRADED) 2024-03-04T16:10:20.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2671755020' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:20.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:20.959 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:20.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:21.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:21.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 73 16 107403 107387 91.9241 63.5625 0.0226943 0.0108714 2024-03-04T16:10:21.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:21.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:10:21.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.226 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:21.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:21.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:21.378 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:21.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:21.534 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:21.636 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:21.671 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:21.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:21.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.809 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:21.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:21.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:21.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:21.999 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:22.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:22.063 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:10:22.063 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:10:22.063 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 3, 4, 5, 6, 7] 2024-03-04T16:10:22.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd out 6 2024-03-04T16:10:22.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 74 16 108438 108422 91.5559 64.6875 0.0128171 0.0109177 2024-03-04T16:10:22.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:22.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:22.237 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:22.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:22.387 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:22.435 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:10:22.435 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:10:22.435 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:10:22.435 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:10:22.435 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:10:22.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:22.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:22.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:22.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:22.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:22.824 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:22.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:22.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:22.969 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:22.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:23.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:23.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 75 16 109521 109505 91.2376 67.6875 0.00632234 0.0109552 2024-03-04T16:10:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:23 smithi176 ceph-mon[33619]: pgmap v325: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 682 B/s rd, 39 MiB/s wr, 626 op/s; 18/315444 objects degraded (0.006%); 114 B/s, 6 objects/s recovering 2024-03-04T16:10:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:23.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.111 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:23.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:23.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:23.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:23.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[29779]: pgmap v325: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 682 B/s rd, 39 MiB/s wr, 626 op/s; 18/315444 objects degraded (0.006%); 114 B/s, 6 objects/s recovering 2024-03-04T16:10:23.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:23.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:23.308 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[38113]: pgmap v325: 148 pgs: 4 remapped+peering, 4 activating+degraded, 14 activating, 4 peering, 122 active+clean; 6.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 682 B/s rd, 39 MiB/s wr, 626 op/s; 18/315444 objects degraded (0.006%); 114 B/s, 6 objects/s recovering 2024-03-04T16:10:23.308 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:23.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:23.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:23.400 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:23.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:10:23.487 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:23.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.540 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:23.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:23.622 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:23.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.693 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:23.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:23.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:23.826 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:23.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:23.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:10:23.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:23.993 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:23.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:24.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 76 16 110565 110549 90.8956 65.25 0.0122291 0.0109968 2024-03-04T16:10:24.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:24.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.138 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:24.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:24.225 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:24.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:24.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.269 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:24.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:24.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:24.406 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:24.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:24.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:24.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:24.689 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:24.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.837 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:24.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:24.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:24.977 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:24.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:25.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:25.087 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 77 16 111612 111596 90.5649 65.4375 0.00564649 0.0110356 2024-03-04T16:10:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.106 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:25.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:25.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[38113]: pgmap v326: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 38551/327942 objects misplaced (11.755%); 20 MiB/s, 335 objects/s recovering 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[38113]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[38113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[38113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded) 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[29779]: pgmap v326: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 38551/327942 objects misplaced (11.755%); 20 MiB/s, 335 objects/s recovering 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[29779]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[29779]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-03-04T16:10:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:25 smithi038 ceph-mon[29779]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded) 2024-03-04T16:10:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:25.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:25.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:25 smithi176 ceph-mon[33619]: pgmap v326: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 38551/327942 objects misplaced (11.755%); 20 MiB/s, 335 objects/s recovering 2024-03-04T16:10:25.266 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:25 smithi176 ceph-mon[33619]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:25.267 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:25 smithi176 ceph-mon[33619]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 1 pg peering) 2024-03-04T16:10:25.267 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:25 smithi176 ceph-mon[33619]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 18/315444 objects degraded (0.006%), 4 pgs degraded) 2024-03-04T16:10:25.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:25.399 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:25.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:25.537 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:25.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.678 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:25.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24051 7.7 7:efc91ac2:::benchmark_data_smithi038_77863_object24050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 328.253162707, 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.133279604, 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24051 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493321+0000", 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3069999999999999e-06 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493378+0000", 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7166000000000001e-05 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493391+0000", 2024-03-04T16:10:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2524000000000001e-05 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493398+0000", 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7810000000000005e-06 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493453+0000", 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4936999999999999e-05 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493576+0000", 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122769 2024-03-04T16:10:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493722+0000", 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014567700000000001 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.494623+0000", 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090136199999999995 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497155+0000", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025317550000000001 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626475+0000", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12932030999999999 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626541+0000", 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5500999999999995e-05 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626599+0000", 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8514999999999998e-05 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24060 7.c 7:300854cb:::benchmark_data_smithi038_77863_object24059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 328.25055248899997, 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13071738699999999, 2024-03-04T16:10:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24060 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495931+0000", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2669999999999999e-06 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496021+0000", 2024-03-04T16:10:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0263e-05 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496036+0000", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4617000000000001e-05 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496042+0000", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0730000000000001e-06 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496109+0000", 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6859999999999993e-05 2024-03-04T16:10:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496221+0000", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112287 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496318+0000", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6948999999999996e-05 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497400+0000", 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010824020000000001 2024-03-04T16:10:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497825+0000", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042419400000000003 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626496+0000", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12867089500000001 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626545+0000", 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9505000000000001e-05 2024-03-04T16:10:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626647+0000", 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102075 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24067 7.c 7:341347ef:::benchmark_data_smithi038_77863_object24066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 328.24709045899999, 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128935775, 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24067 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499394+0000", 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.99e-06 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499451+0000", 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6966999999999999e-05 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499523+0000", 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2533000000000005e-05 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499531+0000", 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9370000000000004e-06 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499584+0000", 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2679999999999997e-05 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499708+0000", 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012427799999999999 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499838+0000", 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129534 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.500954+0000", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001116548 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.501233+0000", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027919700000000002 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628217+0000", 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.126983873 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628271+0000", 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3789000000000003e-05 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628328+0000", 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6449000000000001e-05 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83617 7.d 7:b5cd7487:::benchmark_data_smithi038_77863_object83616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 285.46663382700001, 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11925000199999999, 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83617 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279850+0000", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8840000000000001e-06 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279911+0000", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1161999999999995e-05 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279927+0000", 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5259000000000001e-05 2024-03-04T16:10:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279935+0000", 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5909999999999996e-06 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280005+0000", 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0185000000000006e-05 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280258+0000", 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025285800000000001 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280462+0000", 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020391100000000001 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.286072+0000", 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056095620000000002 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.288597+0000", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025255910000000002 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.398999+0000", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.110401524 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.399039+0000", 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9724000000000001e-05 2024-03-04T16:10:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.399098+0000", 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9750999999999998e-05 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83619 7.d 7:bd3820de:::benchmark_data_smithi038_77863_object83618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 285.46577114000002, 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.119792248, 2024-03-04T16:10:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83619 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280713+0000", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.852e-06 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280746+0000", 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3065000000000003e-05 2024-03-04T16:10:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280758+0000", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1905e-05 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280769+0000", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0851e-05 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283138+0000", 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023693049999999999 2024-03-04T16:10:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283302+0000", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000164069 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283457+0000", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015461000000000001 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.286240+0000", 2024-03-04T16:10:25.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027835030000000001 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.288696+0000", 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024562360000000001 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400382+0000", 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.111685189 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400445+0000", 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3618e-05 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400503+0000", 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8044999999999997e-05 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84320 7.c 7:3115064a:::benchmark_data_smithi038_77863_object84319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:10:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 284.55944371200002, 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.152879031, 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84320 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:10:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187042+0000", 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.994e-06 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187094+0000", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2120000000000002e-05 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187111+0000", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7541999999999999e-05 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187120+0000", 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7110000000000001e-06 2024-03-04T16:10:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187189+0000", 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9076000000000001e-05 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187372+0000", 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183313 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:10:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187533+0000", 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016118799999999999 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190831+0000", 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032979760000000002 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.191323+0000", 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049143600000000004 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339800+0000", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14847677000000001 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339865+0000", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.533e-05 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339918+0000", 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2574999999999997e-05 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84340 7.7 7:e70ccc3e:::benchmark_data_smithi038_77863_object84339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 284.52829966100001, 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14684087500000001, 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84340 2024-03-04T16:10:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218185+0000", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9860000000000001e-06 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218225+0000", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0855000000000001e-05 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218236+0000", 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.097e-05 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218243+0000", 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9380000000000003e-06 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218314+0000", 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1088999999999996e-05 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218446+0000", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131662 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218589+0000", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143261 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.220839+0000", 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002249282 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.221181+0000", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034215299999999999 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.364867+0000", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.143686181 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.364938+0000", 2024-03-04T16:10:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0876000000000004e-05 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365023+0000", 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5622000000000001e-05 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84341 7.c 7:36ea4711:::benchmark_data_smithi038_77863_object84340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 284.518627557, 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121028788, 2024-03-04T16:10:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84341 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227856+0000", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3340000000000001e-06 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227897+0000", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0918999999999999e-05 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227906+0000", 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4099999999999997e-06 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227912+0000", 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5099999999999998e-06 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227961+0000", 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9387e-05 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228096+0000", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013440899999999999 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228252+0000", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015648800000000001 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.230424+0000", 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021721790000000002 2024-03-04T16:10:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.231177+0000", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00075235999999999999 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348763+0000", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.117586044 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348830+0000", 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7508000000000005e-05 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348883+0000", 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3239999999999998e-05 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84342 7.7 7:ee3774c8:::benchmark_data_smithi038_77863_object84341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 284.51803407599999, 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.136723498, 2024-03-04T16:10:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84342 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228449+0000", 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.08e-06 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228512+0000", 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2354000000000006e-05 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228521+0000", 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7049999999999994e-06 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228528+0000", 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9439999999999999e-06 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228582+0000", 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3842999999999999e-05 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228698+0000", 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116078 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228827+0000", 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012882400000000001 2024-03-04T16:10:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.231154+0000", 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023271849999999998 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232107+0000", 2024-03-04T16:10:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095312599999999999 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365079+0000", 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13297209800000001 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365133+0000", 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3532999999999997e-05 2024-03-04T16:10:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365172+0000", 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8727999999999999e-05 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84344 7.7 7:ef5cf9b4:::benchmark_data_smithi038_77863_object84343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:10:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 284.51398498700001, 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132825691, 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84344 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232499+0000", 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6390000000000001e-06 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232550+0000", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1088000000000003e-05 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232559+0000", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3190000000000001e-06 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232565+0000", 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2229999999999998e-06 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232635+0000", 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0155999999999997e-05 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232718+0000", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3217e-05 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232809+0000", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1291000000000004e-05 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.235006+0000", 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021970459999999998 2024-03-04T16:10:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.236595+0000", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001589177 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365224+0000", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12862828400000001 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365278+0000", 2024-03-04T16:10:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4613e-05 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365323+0000", 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4638000000000002e-05 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:138242 7.7 7:e7af825f:::benchmark_data_smithi038_77863_object19974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 228.237769335, 2024-03-04T16:10:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13032482200000001, 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 138242 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508714+0000", 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.26e-06 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508720+0000", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3460000000000001e-06 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508729+0000", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8689999999999999e-06 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508737+0000", 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9780000000000005e-06 2024-03-04T16:10:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508801+0000", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3350999999999997e-05 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508997+0000", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019587599999999999 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.509104+0000", 2024-03-04T16:10:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107469 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.514111+0000", 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050073970000000002 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.514864+0000", 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00075243800000000002 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.638975+0000", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.124111007 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.639007+0000", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2280000000000003e-05 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.639038+0000", 2024-03-04T16:10:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0550999999999999e-05 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:138252 7.7 7:ef9db293:::benchmark_data_smithi038_77863_object19984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 228.22694623699999, 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12241484399999999, 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 138252 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:10:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519538+0000", 2024-03-04T16:10:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.841e-06 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519543+0000", 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6400000000000002e-06 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519555+0000", 2024-03-04T16:10:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1358999999999999e-05 2024-03-04T16:10:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519563+0000", 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7759999999999997e-06 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519632+0000", 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9832000000000005e-05 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519781+0000", 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014851 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519910+0000", 2024-03-04T16:10:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012902399999999999 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.523610+0000", 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037004020000000002 2024-03-04T16:10:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.524514+0000", 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090372900000000003 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641877+0000", 2024-03-04T16:10:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11736239 2024-03-04T16:10:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641918+0000", 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1087999999999997e-05 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641951+0000", 2024-03-04T16:10:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3253000000000002e-05 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159707 7.d 7:b4f69bdc:::benchmark_data_smithi038_77863_object41439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 205.108585371, 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140372144, 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159707 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637899+0000", 2024-03-04T16:10:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.792e-06 2024-03-04T16:10:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637904+0000", 2024-03-04T16:10:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6570000000000002e-06 2024-03-04T16:10:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637918+0000", 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4014e-05 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637927+0000", 2024-03-04T16:10:25.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2390000000000002e-06 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637988+0000", 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1268999999999999e-05 2024-03-04T16:10:25.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638102+0000", 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114027 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638185+0000", 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3429000000000004e-05 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.640863+0000", 2024-03-04T16:10:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026778000000000001 2024-03-04T16:10:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.768307+0000", 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12744344799999999 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778170+0000", 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098631480000000004 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778221+0000", 2024-03-04T16:10:25.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1202000000000002e-05 2024-03-04T16:10:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778269+0000", 2024-03-04T16:10:25.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8118999999999998e-05 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.853 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159712 7.d 7:b9855caa:::benchmark_data_smithi038_77863_object41444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:10:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 205.10377847000001, 2024-03-04T16:10:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135708674, 2024-03-04T16:10:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.854 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159712 2024-03-04T16:10:25.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:10:25.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642705+0000", 2024-03-04T16:10:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3430000000000001e-06 2024-03-04T16:10:25.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642710+0000", 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4780000000000002e-06 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642718+0000", 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8410000000000006e-06 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642724+0000", 2024-03-04T16:10:25.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2669999999999997e-06 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642760+0000", 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6625000000000003e-05 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.861 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642844+0000", 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3880000000000003e-05 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642918+0000", 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3978000000000002e-05 2024-03-04T16:10:25.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.647940+0000", 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050219000000000001 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.768391+0000", 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12045135799999999 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778324+0000", 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099328090000000008 2024-03-04T16:10:25.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778374+0000", 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0189999999999999e-05 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778412+0000", 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8005e-05 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159713 7.7 7:e219d9b5:::benchmark_data_smithi038_77863_object41445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 205.10374602499999, 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13068476400000001, 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159713 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642737+0000", 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0500000000000003e-07 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642740+0000", 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7140000000000002e-06 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642745+0000", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6749999999999999e-06 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642749+0000", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9600000000000002e-06 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642810+0000", 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0588999999999999e-05 2024-03-04T16:10:25.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642920+0000", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109808 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643036+0000", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116013 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645568+0000", 2024-03-04T16:10:25.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025322019999999999 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.772612+0000", 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12704433300000001 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773350+0000", 2024-03-04T16:10:25.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00073744000000000001 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773386+0000", 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6205000000000003e-05 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773421+0000", 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5219999999999998e-05 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.872 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159717 7.d 7:b0b31d2c:::benchmark_data_smithi038_77863_object41449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 205.10172694100001, 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13377419199999999, 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.873 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159717 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:10:25.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:25.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644757+0000", 2024-03-04T16:10:25.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.333e-06 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644761+0000", 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2570000000000001e-06 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644771+0000", 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0434999999999999e-05 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644777+0000", 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6409999999999999e-06 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645495+0000", 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00071810600000000004 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645614+0000", 2024-03-04T16:10:25.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011852499999999999 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645732+0000", 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118632 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.648042+0000", 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002309644 2024-03-04T16:10:25.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776738+0000", 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12869599000000001 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778452+0000", 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017144409999999999 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778496+0000", 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3905000000000001e-05 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778529+0000", 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3283e-05 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163737 7.7 7:e92a9127:::benchmark_data_smithi038_77863_object45469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.886 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 200.57033328599999, 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1836797790000002, 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163737 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:10:25.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176151+0000", 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7600000000000001e-06 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176156+0000", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9230000000000001e-06 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176167+0000", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1066e-05 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176173+0000", 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.319e-06 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176237+0000", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3984000000000003e-05 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176404+0000", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733899999999999 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176531+0000", 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012694900000000001 2024-03-04T16:10:25.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.179756+0000", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003224401 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.193019+0000", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013263138000000001 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204566+0000", 2024-03-04T16:10:25.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011546910000000001 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204579+0000", 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2941e-05 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215396+0000", 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.010817297 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215422+0000", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5999999999999998e-05 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.359741+0000", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.144318645 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.359829+0000", 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8107000000000002e-05 2024-03-04T16:10:25.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163746 7.7 7:ee84827a:::benchmark_data_smithi038_77863_object45478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 200.563408853, 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.177985804, 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.894 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163746 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183076+0000", 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3640000000000002e-06 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183082+0000", 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5080000000000002e-06 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183097+0000", 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183105+0000", 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4190000000000002e-06 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183161+0000", 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5736000000000003e-05 2024-03-04T16:10:25.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183296+0000", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134661 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183415+0000", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118998 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.189322+0000", 2024-03-04T16:10:25.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059075170000000001 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204612+0000", 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015290056 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204626+0000", 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3361e-05 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215543+0000", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0109174489999999 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215555+0000", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1612e-05 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360112+0000", 2024-03-04T16:10:25.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1445573200000001 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.361059+0000", 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000947039 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1414 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93955426370688 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:10:25.901 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 200.55876061699999, 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.048487942, 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1414 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187724+0000", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5770000000000001e-06 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187731+0000", 2024-03-04T16:10:25.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6370000000000002e-06 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187750+0000", 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9174999999999999e-05 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187761+0000", 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1079e-05 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187835+0000", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3913999999999997e-05 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187844+0000", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6999999999999997e-06 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.201860+0000", 2024-03-04T16:10:25.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014015929 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.201870+0000", 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0411000000000001e-05 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.228681+0000", 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.026810666 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.229187+0000", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050616300000000001 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.233029+0000", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038424380000000001 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.233132+0000", 2024-03-04T16:10:25.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010299299999999999 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.236210+0000", 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030772600000000001 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.908 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163754 7.7 7:e3a25a20:::benchmark_data_smithi038_77863_object45486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 200.53443681499999, 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.189298231, 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163754 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:10:25.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212047+0000", 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.007e-06 2024-03-04T16:10:25.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212054+0000", 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3670000000000002e-06 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:10:25.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212064+0000", 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0253e-05 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212072+0000", 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1310000000000006e-06 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.912 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212135+0000", 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2322000000000004e-05 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212146+0000", 2024-03-04T16:10:25.913 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1117999999999999e-05 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215584+0000", 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.00343866 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:10:25.914 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215593+0000", 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-06 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360201+0000", 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1446078790000001 2024-03-04T16:10:25.915 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360324+0000", 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122609 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,3", 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360772+0000", 2024-03-04T16:10:25.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044833200000000002 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.371865+0000", 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011092328 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.383538+0000", 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011673174 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.394054+0000", 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010515871 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401133+0000", 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070789609999999999 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401200+0000", 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7162000000000005e-05 2024-03-04T16:10:25.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401344+0000", 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014398699999999999 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.920 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:10:25.921 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:10:25.921 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:10:25.921 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:25.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:26.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:26.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:26.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 78 16 112496 112480 90.1122 55.25 0.0096068 0.0110924 2024-03-04T16:10:26.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:26.206 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:26.211 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:10:26.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:26.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:26.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:26 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/114448269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:26 smithi038 ceph-mon[38113]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.494 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:26.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:26 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/114448269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:26 smithi038 ceph-mon[29779]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:26 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/114448269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:26 smithi176 ceph-mon[33619]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:26.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:26.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:26.772 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:26.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:26.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:26.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:27.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:27.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:27.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 79 16 113600 113584 89.8449 69 0.00809731 0.0111258 2024-03-04T16:10:27.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:10:27.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:27.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:27.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[38113]: pgmap v327: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 991 op/s; 38551/327942 objects misplaced (11.755%); 18 MiB/s, 292 objects/s recovering 2024-03-04T16:10:27.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:27.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[38113]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:27.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[38113]: osdmap e193: 8 total, 8 up, 7 in 2024-03-04T16:10:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[29779]: pgmap v327: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 991 op/s; 38551/327942 objects misplaced (11.755%); 18 MiB/s, 292 objects/s recovering 2024-03-04T16:10:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[29779]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:27 smithi038 ceph-mon[29779]: osdmap e193: 8 total, 8 up, 7 in 2024-03-04T16:10:27.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:27 smithi176 ceph-mon[33619]: pgmap v327: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 62 MiB/s wr, 991 op/s; 38551/327942 objects misplaced (11.755%); 18 MiB/s, 292 objects/s recovering 2024-03-04T16:10:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:27 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:27 smithi176 ceph-mon[33619]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:10:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:27 smithi176 ceph-mon[33619]: osdmap e193: 8 total, 8 up, 7 in 2024-03-04T16:10:27.377 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:27.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:27.510 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:27.516 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:27.523 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:27.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.667 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:27.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:27.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:27.813 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:27.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:27.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:27.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:28.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:28.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:10:28.088711+0000 min lat: 0.00119552 max lat: 2.36607 avg lat: 0.0111667 2024-03-04T16:10:28.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:10:28.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 80 16 114613 114597 89.5132 63.3125 0.0105709 0.0111667 2024-03-04T16:10:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:28.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:28.193 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:28.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:28.234 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.6. 2024-03-04T16:10:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:10:28.018+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e193 definitely_dead 0 2024-03-04T16:10:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1284218961' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[29779]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[29779]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-04T16:10:28.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1284218961' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[38113]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.254 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:28 smithi038 ceph-mon[38113]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-04T16:10:28.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:28.357 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:28.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:28.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:28.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:28 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1284218961' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:28 smithi176 ceph-mon[33619]: from='client.35138 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-04T16:10:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:28 smithi176 ceph-mon[33619]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-04T16:10:28.617 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:28.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:28.747 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:28.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:28.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:28.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:28.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:29.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:29.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:10:29.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 81 16 114793 114777 88.547 11.25 0.004672 0.0111711 2024-03-04T16:10:29.108 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:29.138 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:29.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:29.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:29.273 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:29.309 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:29 smithi176 ceph-mon[33619]: pgmap v329: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 569 B/s rd, 29 MiB/s wr, 463 op/s; 38551/327942 objects misplaced (11.755%); 16 MiB/s, 256 objects/s recovering 2024-03-04T16:10:29.310 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:29 smithi176 ceph-mon[33619]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-04T16:10:29.310 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:29 smithi176 ceph-mon[33619]: osdmap e194: 8 total, 8 up, 6 in 2024-03-04T16:10:29.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:29.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:29.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[29779]: pgmap v329: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 569 B/s rd, 29 MiB/s wr, 463 op/s; 38551/327942 objects misplaced (11.755%); 16 MiB/s, 256 objects/s recovering 2024-03-04T16:10:29.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[29779]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-04T16:10:29.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[29779]: osdmap e194: 8 total, 8 up, 6 in 2024-03-04T16:10:29.491 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[38113]: pgmap v329: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 569 B/s rd, 29 MiB/s wr, 463 op/s; 38551/327942 objects misplaced (11.755%); 16 MiB/s, 256 objects/s recovering 2024-03-04T16:10:29.491 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[38113]: from='client.35138 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-04T16:10:29.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:29 smithi038 ceph-mon[38113]: osdmap e194: 8 total, 8 up, 6 in 2024-03-04T16:10:29.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:29.542 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:29.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:29.617 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:29.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:29.675 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:29.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:29.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:29.807 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:29.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:29.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:29.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:30.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.068 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:30.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:30.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 82 16 114809 114793 87.4795 1 0.00812278 0.0112332 2024-03-04T16:10:30.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:30.194 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:30.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.338 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:30.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:30.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[29779]: osdmap e195: 8 total, 8 up, 6 in 2024-03-04T16:10:30.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[29779]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7108 2024-03-04T16:10:30.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[29779]: 8.d starting backfill to osd.1 from (0'0,0'0] MAX to 193'7181 2024-03-04T16:10:30.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[29779]: pgmap v332: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 858 op/s; 37515/343380 objects misplaced (10.925%); 29 MiB/s, 466 objects/s recovering 2024-03-04T16:10:30.357 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[38113]: osdmap e195: 8 total, 8 up, 6 in 2024-03-04T16:10:30.357 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[38113]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7108 2024-03-04T16:10:30.357 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[38113]: 8.d starting backfill to osd.1 from (0'0,0'0] MAX to 193'7181 2024-03-04T16:10:30.357 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:30 smithi038 ceph-mon[38113]: pgmap v332: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 858 op/s; 37515/343380 objects misplaced (10.925%); 29 MiB/s, 466 objects/s recovering 2024-03-04T16:10:30.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:30.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:30.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:30 smithi176 ceph-mon[33619]: osdmap e195: 8 total, 8 up, 6 in 2024-03-04T16:10:30.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:30 smithi176 ceph-mon[33619]: 8.4 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7108 2024-03-04T16:10:30.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:30 smithi176 ceph-mon[33619]: 8.d starting backfill to osd.1 from (0'0,0'0] MAX to 193'7181 2024-03-04T16:10:30.541 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:30 smithi176 ceph-mon[33619]: pgmap v332: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 858 op/s; 37515/343380 objects misplaced (10.925%); 29 MiB/s, 466 objects/s recovering 2024-03-04T16:10:30.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:30.615 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:30.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:30.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:30.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:30.943 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:30.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:31.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.083 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:31.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:31.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 83 16 115454 115438 86.9112 40.3125 0.00594279 0.011491 2024-03-04T16:10:31.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:31.212 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:31.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:31.349 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:31.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:31.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:31.487 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: osdmap e196: 8 total, 8 up, 6 in 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 193'7296 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: 8.6 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7127 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7092 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 193'7117 2024-03-04T16:10:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[38113]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7078 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: osdmap e196: 8 total, 8 up, 6 in 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 193'7296 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: 8.6 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7127 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7092 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 193'7117 2024-03-04T16:10:31.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:31 smithi038 ceph-mon[29779]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7078 2024-03-04T16:10:31.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: osdmap e196: 8 total, 8 up, 6 in 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 193'7296 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: 8.6 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7127 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: 8.3 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7092 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: 8.7 starting backfill to osd.4 from (0'0,0'0] MAX to 193'7117 2024-03-04T16:10:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:31 smithi176 ceph-mon[33619]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 193'7078 2024-03-04T16:10:31.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:31.752 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:31.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:31.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:31.832 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:10:31.878 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:10:31.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:31.925 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:10:31.936 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:31.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:31.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:31.973 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:10:32.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:10:32.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:32.070 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:10:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:32.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 84 16 116238 116222 86.4599 49 0.00701 0.0115612 2024-03-04T16:10:32.115 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:10:32.164 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:10:32.171 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:10:32.171 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:10:32.172 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:10:32.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:32.215 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:32.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:32.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:32.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[29779]: osdmap e197: 8 total, 8 up, 6 in 2024-03-04T16:10:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[29779]: pgmap v335: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 37515/343380 objects misplaced (10.925%); 44 MiB/s, 699 objects/s recovering 2024-03-04T16:10:32.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1704606284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[38113]: osdmap e197: 8 total, 8 up, 6 in 2024-03-04T16:10:32.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[38113]: pgmap v335: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 37515/343380 objects misplaced (10.925%); 44 MiB/s, 699 objects/s recovering 2024-03-04T16:10:32.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:32 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1704606284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:32.504 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:32.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:32.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:32 smithi176 ceph-mon[33619]: osdmap e197: 8 total, 8 up, 6 in 2024-03-04T16:10:32.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:32 smithi176 ceph-mon[33619]: pgmap v335: 148 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 142 active+clean; 7.0 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.29k op/s; 37515/343380 objects misplaced (10.925%); 44 MiB/s, 699 objects/s recovering 2024-03-04T16:10:32.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:32 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1704606284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:32.712 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:10:32.712 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:10:32.712 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-03-04T16:10:32.712 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:10:32.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:32.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:32.937 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:32.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:33.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:33.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:33.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 85 16 117096 117080 86.0736 53.625 0.00975143 0.0116127 2024-03-04T16:10:33.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:33.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:33.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:33.293 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:10:33.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.358 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:33.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:33.442 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:33 smithi176 ceph-mon[33619]: osdmap e198: 8 total, 8 up, 6 in 2024-03-04T16:10:33.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:33.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/782107681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.443 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:33 smithi176 ceph-mon[33619]: from='client.35144 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:33.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:10:33.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:33.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:33.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:33.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[29779]: osdmap e198: 8 total, 8 up, 6 in 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/782107681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[29779]: from='client.35144 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[38113]: osdmap e198: 8 total, 8 up, 6 in 2024-03-04T16:10:33.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:33.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/782107681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:33 smithi038 ceph-mon[38113]: from='client.35144 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:10:33.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:33.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:33.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:33.905 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:33.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:33.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:34.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:34.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:34.080 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:10:34.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:10:34.080 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-04T16:10:34.080 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-04T16:10:34.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:10:34.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 86 16 118087 118071 85.7929 61.9375 0.0063866 0.0116468 2024-03-04T16:10:34.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:34.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:34.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:34 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:34.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:34 smithi176 ceph-mon[33619]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:10:34.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:34 smithi176 ceph-mon[33619]: osdmap e199: 8 total, 8 up, 6 in 2024-03-04T16:10:34.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:34 smithi176 ceph-mon[33619]: pgmap v338: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 511 B/s rd, 39 MiB/s wr, 621 op/s; 70488/350835 objects misplaced (20.091%); 92 MiB/s, 1.50k objects/s recovering 2024-03-04T16:10:34.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:34.387 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:34.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:34.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[29779]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:10:34.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[29779]: osdmap e199: 8 total, 8 up, 6 in 2024-03-04T16:10:34.419 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[29779]: pgmap v338: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 511 B/s rd, 39 MiB/s wr, 621 op/s; 70488/350835 objects misplaced (20.091%); 92 MiB/s, 1.50k objects/s recovering 2024-03-04T16:10:34.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:10:34.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[38113]: from='client.35144 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:10:34.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[38113]: osdmap e199: 8 total, 8 up, 6 in 2024-03-04T16:10:34.420 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:34 smithi038 ceph-mon[38113]: pgmap v338: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 511 B/s rd, 39 MiB/s wr, 621 op/s; 70488/350835 objects misplaced (20.091%); 92 MiB/s, 1.50k objects/s recovering 2024-03-04T16:10:34.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:34.523 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:34.600 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:34.625 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:34.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:34.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:34.805 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:34.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:34.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:34.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.000 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:35.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:35.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:10:35.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 87 16 118977 118961 85.4461 55.625 0.0211801 0.0116972 2024-03-04T16:10:35.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.138 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:35.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:35.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:35.289 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:35 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:10:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:35 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:10:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:35 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:35.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.431 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:35.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:10:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:35 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:10:35.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:35.561 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:35.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:35.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:35.710 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:35.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:35.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:35.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:35.860 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:35.860 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:35.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:36.003 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:36.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:36.088 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 88 16 119921 119905 85.1455 59 0.0200801 0.0117396 2024-03-04T16:10:36.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:36.148 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:36.250 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.293 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:36.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:36.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:36.411 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:36.463 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:36.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:10:36.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:36 smithi038 ceph-mon[29779]: pgmap v339: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 392 B/s rd, 30 MiB/s wr, 476 op/s; 70488/350835 objects misplaced (20.091%); 71 MiB/s, 1.15k objects/s recovering 2024-03-04T16:10:36.544 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:36 smithi038 ceph-mon[38113]: pgmap v339: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 392 B/s rd, 30 MiB/s wr, 476 op/s; 70488/350835 objects misplaced (20.091%); 71 MiB/s, 1.15k objects/s recovering 2024-03-04T16:10:36.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:36.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:36 smithi176 ceph-mon[33619]: pgmap v339: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 392 B/s rd, 30 MiB/s wr, 476 op/s; 70488/350835 objects misplaced (20.091%); 71 MiB/s, 1.15k objects/s recovering 2024-03-04T16:10:36.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:10:36.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:36.731 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:36.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:36.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:36.879 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:36.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:36.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:37.026 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:37.059 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:10:37.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 89 16 120811 120795 84.8138 55.625 0.0153805 0.0117852 2024-03-04T16:10:37.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:37.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:37.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:37.293 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:37.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:37.426 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:37.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:37.565 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:37.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.710 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:37.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:37.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:37.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:10:37.888 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:10:37.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:37.990 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:37.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:38.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 90 1 121632 121631 84.452 52.25 0.0141273 0.011836 2024-03-04T16:10:38.091 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:38.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:38.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:38.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:38.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:38.388 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:38.490 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:38.519 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:38.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:38.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:38.780 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-03-04T16:10:37.447684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7651727710,"num_objects":116945,"num_object_clones":0,"num_object_copies":350835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":70488,"num_objects_unfound":0,"num_objects_dirty":116945,"num_whiteouts":0,"num_read":1381,"num_read_kb":1227,"num_write":117362,"num_write_kb":7474269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11252,"num_bytes_recovered":723594514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65628,"ondisk_log_size":65628,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":442,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19419828,"kb_used_data":17844456,"kb_used_omap":9,"kb_used_meta":1575286,"kb_avail":543051084,"statfs":{"total":575970213888,"available":556084310016,"internally_reserved":0,"allocated":18272722944,"data_stored":18270197286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":1613093543},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,2,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":162856960,"num_objects":2485,"num_object_clones":0,"num_object_copies":7455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":32973,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":2485,"num_write_kb":159040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5984,"num_bytes_recovered":387516624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.197926"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263737+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:10:32.263737+0000","last_peered":"2024-03-04T16:10:32.263737+0000","last_clean":"2024-03-04T16:10:32.263737+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:10:32.263737+0000","last_undegraded":"2024-03-04T16:10:32.263737+0000","last_fullsized":"2024-03-04T16:10:32.263737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:44.024543+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.273933+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:10:32.273933+0000","last_peered":"2024-03-04T16:10:32.273933+0000","last_clean":"2024-03-04T16:10:32.273933+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:10:32.273933+0000","last_undegraded":"2024-03-04T16:10:32.273933+0000","last_fullsized":"2024-03-04T16:10:32.273933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":146,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.279293+0000","last_change":"2024-03-04T16:10:29.339688+0000","last_active":"2024-03-04T16:10:32.279293+0000","last_peered":"2024-03-04T16:10:32.279293+0000","last_clean":"2024-03-04T16:10:32.279293+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:10:32.279293+0000","last_undegraded":"2024-03-04T16:10:32.279293+0000","last_fullsized":"2024-03-04T16:10:32.279293+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:34.628299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":334,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263375+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:10:32.263375+0000","last_peered":"2024-03-04T16:10:32.263375+0000","last_clean":"2024-03-04T16:10:32.263375+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:10:32.263375+0000","last_undegraded":"2024-03-04T16:10:32.263375+0000","last_fullsized":"2024-03-04T16:10:32.263375+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.733521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288250+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:10:32.288250+0000","last_peered":"2024-03-04T16:10:32.288250+0000","last_clean":"2024-03-04T16:10:32.288250+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:10:32.288250+0000","last_undegraded":"2024-03-04T16:10:32.288250+0000","last_fullsized":"2024-03-04T16:10:32.288250+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:46.582084+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263408+0000","last_change":"2024-03-04T16:10:29.325452+0000","last_active":"2024-03-04T16:10:32.263408+0000","last_peered":"2024-03-04T16:10:32.263408+0000","last_clean":"2024-03-04T16:10:32.263408+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:10:32.263408+0000","last_undegraded":"2024-03-04T16:10:32.263408+0000","last_fullsized":"2024-03-04T16:10:32.263408+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":302,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288627+0000","last_change":"2024-03-04T16:10:29.251726+0000","last_active":"2024-03-04T16:10:32.288627+0000","last_peered":"2024-03-04T16:10:32.288627+0000","last_clean":"2024-03-04T16:10:32.288627+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:10:32.288627+0000","last_undegraded":"2024-03-04T16:10:32.288627+0000","last_fullsized":"2024-03-04T16:10:32.288627+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":385,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.280528+0000","last_change":"2024-03-04T16:10:17.503786+0000","last_active":"2024-03-04T16:10:32.280528+0000","last_peered":"2024-03-04T16:10:32.280528+0000","last_clean":"2024-03-04T16:10:32.280528+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:10:32.280528+0000","last_undegraded":"2024-03-04T16:10:32.280528+0000","last_fullsized":"2024-03-04T16:10:32.280528+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":142,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275008+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:10:32.275008+0000","last_peered":"2024-03-04T16:10:32.275008+0000","last_clean":"2024-03-04T16:10:32.275008+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:10:32.275008+0000","last_undegraded":"2024-03-04T16:10:32.275008+0000","last_fullsized":"2024-03-04T16:10:32.275008+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:35.559365+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":289,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266500+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:10:32.266500+0000","last_peered":"2024-03-04T16:10:32.266500+0000","last_clean":"2024-03-04T16:10:32.266500+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:10:32.266500+0000","last_undegraded":"2024-03-04T16:10:32.266500+0000","last_fullsized":"2024-03-04T16:10:32.266500+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":311,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267792+0000","last_change":"2024-03-04T16:10:29.368250+0000","last_active":"2024-03-04T16:10:32.267792+0000","last_peered":"2024-03-04T16:10:32.267792+0000","last_clean":"2024-03-04T16:10:32.267792+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:10:32.267792+0000","last_undegraded":"2024-03-04T16:10:32.267792+0000","last_fullsized":"2024-03-04T16:10:32.267792+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":346,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263866+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:10:32.263866+0000","last_peered":"2024-03-04T16:10:32.263866+0000","last_clean":"2024-03-04T16:10:32.263866+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:10:32.263866+0000","last_undegraded":"2024-03-04T16:10:32.263866+0000","last_fullsized":"2024-03-04T16:10:32.263866+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":301,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267532+0000","last_change":"2024-03-04T16:10:02.242059+0000","last_active":"2024-03-04T16:10:32.267532+0000","last_peered":"2024-03-04T16:10:32.267532+0000","last_clean":"2024-03-04T16:10:32.267532+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:10:32.267532+0000","last_undegraded":"2024-03-04T16:10:32.267532+0000","last_fullsized":"2024-03-04T16:10:32.267532+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.241922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.241922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.249200+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268067+0000","last_change":"2024-03-04T16:10:08.741643+0000","last_active":"2024-03-04T16:10:32.268067+0000","last_peered":"2024-03-04T16:10:32.268067+0000","last_clean":"2024-03-04T16:10:32.268067+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:10:32.268067+0000","last_undegraded":"2024-03-04T16:10:32.268067+0000","last_fullsized":"2024-03-04T16:10:32.268067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:08.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:08.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:15.259742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267390+0000","last_change":"2024-03-04T16:10:00.097245+0000","last_active":"2024-03-04T16:10:32.267390+0000","last_peered":"2024-03-04T16:10:32.267390+0000","last_clean":"2024-03-04T16:10:32.267390+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:10:32.267390+0000","last_undegraded":"2024-03-04T16:10:32.267390+0000","last_fullsized":"2024-03-04T16:10:32.267390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.097027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.097027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:09.191697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":330,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266583+0000","last_change":"2024-03-04T16:10:17.490328+0000","last_active":"2024-03-04T16:10:32.266583+0000","last_peered":"2024-03-04T16:10:32.266583+0000","last_clean":"2024-03-04T16:10:32.266583+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:10:32.266583+0000","last_undegraded":"2024-03-04T16:10:32.266583+0000","last_fullsized":"2024-03-04T16:10:32.266583+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264553+0000","last_change":"2024-03-04T16:10:29.330112+0000","last_active":"2024-03-04T16:10:32.264553+0000","last_peered":"2024-03-04T16:10:32.264553+0000","last_clean":"2024-03-04T16:10:32.264553+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:10:32.264553+0000","last_undegraded":"2024-03-04T16:10:32.264553+0000","last_fullsized":"2024-03-04T16:10:32.264553+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275426+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:10:32.275426+0000","last_peered":"2024-03-04T16:10:32.275426+0000","last_clean":"2024-03-04T16:10:32.275426+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:10:32.275426+0000","last_undegraded":"2024-03-04T16:10:32.275426+0000","last_fullsized":"2024-03-04T16:10:32.275426+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.276153+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:10:32.276153+0000","last_peered":"2024-03-04T16:10:32.276153+0000","last_clean":"2024-03-04T16:10:32.276153+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:10:32.276153+0000","last_undegraded":"2024-03-04T16:10:32.276153+0000","last_fullsized":"2024-03-04T16:10:32.276153+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.697821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":86,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274141+0000","last_change":"2024-03-04T16:10:17.515216+0000","last_active":"2024-03-04T16:10:32.274141+0000","last_peered":"2024-03-04T16:10:32.274141+0000","last_clean":"2024-03-04T16:10:32.274141+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:10:32.274141+0000","last_undegraded":"2024-03-04T16:10:32.274141+0000","last_fullsized":"2024-03-04T16:10:32.274141+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:10:01.944401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.944401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:10.499604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.293440+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:10:32.293440+0000","last_peered":"2024-03-04T16:10:32.293440+0000","last_clean":"2024-03-04T16:10:32.293440+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:10:32.293440+0000","last_undegraded":"2024-03-04T16:10:32.293440+0000","last_fullsized":"2024-03-04T16:10:32.293440+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.509351+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265077+0000","last_change":"2024-03-04T16:10:17.459301+0000","last_active":"2024-03-04T16:10:32.265077+0000","last_peered":"2024-03-04T16:10:32.265077+0000","last_clean":"2024-03-04T16:10:32.265077+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:10:32.265077+0000","last_undegraded":"2024-03-04T16:10:32.265077+0000","last_fullsized":"2024-03-04T16:10:32.265077+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.804222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.804222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.878528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263936+0000","last_change":"2024-03-04T16:10:17.481021+0000","last_active":"2024-03-04T16:10:32.263936+0000","last_peered":"2024-03-04T16:10:32.263936+0000","last_clean":"2024-03-04T16:10:32.263936+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:10:32.263936+0000","last_undegraded":"2024-03-04T16:10:32.263936+0000","last_fullsized":"2024-03-04T16:10:32.263936+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:41.867486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288267+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:10:32.288267+0000","last_peered":"2024-03-04T16:10:32.288267+0000","last_clean":"2024-03-04T16:10:32.288267+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:10:32.288267+0000","last_undegraded":"2024-03-04T16:10:32.288267+0000","last_fullsized":"2024-03-04T16:10:32.288267+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268570+0000","last_change":"2024-03-04T16:10:17.485038+0000","last_active":"2024-03-04T16:10:32.268570+0000","last_peered":"2024-03-04T16:10:32.268570+0000","last_clean":"2024-03-04T16:10:32.268570+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:10:32.268570+0000","last_undegraded":"2024-03-04T16:10:32.268570+0000","last_fullsized":"2024-03-04T16:10:32.268570+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:09:01.792870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.792870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":217,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265513+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:10:32.265513+0000","last_peered":"2024-03-04T16:10:32.265513+0000","last_clean":"2024-03-04T16:10:32.265513+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:10:32.265513+0000","last_undegraded":"2024-03-04T16:10:32.265513+0000","last_fullsized":"2024-03-04T16:10:32.265513+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.314870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274257+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:10:32.274257+0000","last_peered":"2024-03-04T16:10:32.274257+0000","last_clean":"2024-03-04T16:10:32.274257+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:10:32.274257+0000","last_undegraded":"2024-03-04T16:10:32.274257+0000","last_fullsized":"2024-03-04T16:10:32.274257+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":154,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.287078+0000","last_change":"2024-03-04T16:09:58.487155+0000","last_active":"2024-03-04T16:10:32.287078+0000","last_peered":"2024-03-04T16:10:32.287078+0000","last_clean":"2024-03-04T16:10:32.287078+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:10:32.287078+0000","last_undegraded":"2024-03-04T16:10:32.287078+0000","last_fullsized":"2024-03-04T16:10:32.287078+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:58.487024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.487024+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:24.715565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275563+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:10:32.275563+0000","last_peered":"2024-03-04T16:10:32.275563+0000","last_clean":"2024-03-04T16:10:32.275563+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:10:32.275563+0000","last_undegraded":"2024-03-04T16:10:32.275563+0000","last_fullsized":"2024-03-04T16:10:32.275563+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.287339+0000","last_change":"2024-03-04T16:10:17.479793+0000","last_active":"2024-03-04T16:10:32.287339+0000","last_peered":"2024-03-04T16:10:32.287339+0000","last_clean":"2024-03-04T16:10:32.287339+0000","last_became_active":"2024-03-04T16:10:17.478936+0000","last_became_peered":"2024-03-04T16:10:17.478936+0000","last_unstale":"2024-03-04T16:10:32.287339+0000","last_undegraded":"2024-03-04T16:10:32.287339+0000","last_fullsized":"2024-03-04T16:10:32.287339+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.697762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281558+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:10:32.281558+0000","last_peered":"2024-03-04T16:10:32.281558+0000","last_clean":"2024-03-04T16:10:32.281558+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:10:32.281558+0000","last_undegraded":"2024-03-04T16:10:32.281558+0000","last_fullsized":"2024-03-04T16:10:32.281558+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":68,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266931+0000","last_change":"2024-03-04T16:10:29.384325+0000","last_active":"2024-03-04T16:10:32.266931+0000","last_peered":"2024-03-04T16:10:32.266931+0000","last_clean":"2024-03-04T16:10:32.266931+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:10:32.266931+0000","last_undegraded":"2024-03-04T16:10:32.266931+0000","last_fullsized":"2024-03-04T16:10:32.266931+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267981+0000","last_change":"2024-03-04T16:10:17.457894+0000","last_active":"2024-03-04T16:10:32.267981+0000","last_peered":"2024-03-04T16:10:32.267981+0000","last_clean":"2024-03-04T16:10:32.267981+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:10:32.267981+0000","last_undegraded":"2024-03-04T16:10:32.267981+0000","last_fullsized":"2024-03-04T16:10:32.267981+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:09:19.779373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.475501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263598+0000","last_change":"2024-03-04T16:10:29.238069+0000","last_active":"2024-03-04T16:10:32.263598+0000","last_peered":"2024-03-04T16:10:32.263598+0000","last_clean":"2024-03-04T16:10:32.263598+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:10:32.263598+0000","last_undegraded":"2024-03-04T16:10:32.263598+0000","last_fullsized":"2024-03-04T16:10:32.263598+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.279207+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:10:32.279207+0000","last_peered":"2024-03-04T16:10:32.279207+0000","last_clean":"2024-03-04T16:10:32.279207+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:10:32.279207+0000","last_undegraded":"2024-03-04T16:10:32.279207+0000","last_fullsized":"2024-03-04T16:10:32.279207+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.697793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281669+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:10:32.281669+0000","last_peered":"2024-03-04T16:10:32.281669+0000","last_clean":"2024-03-04T16:10:32.281669+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:10:32.281669+0000","last_undegraded":"2024-03-04T16:10:32.281669+0000","last_fullsized":"2024-03-04T16:10:32.281669+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:58.997746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288517+0000","last_change":"2024-03-04T16:10:29.250985+0000","last_active":"2024-03-04T16:10:32.288517+0000","last_peered":"2024-03-04T16:10:32.288517+0000","last_clean":"2024-03-04T16:10:32.288517+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:10:32.288517+0000","last_undegraded":"2024-03-04T16:10:32.288517+0000","last_fullsized":"2024-03-04T16:10:32.288517+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":296,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267210+0000","last_change":"2024-03-04T16:10:29.248681+0000","last_active":"2024-03-04T16:10:32.267210+0000","last_peered":"2024-03-04T16:10:32.267210+0000","last_clean":"2024-03-04T16:10:32.267210+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:10:32.267210+0000","last_undegraded":"2024-03-04T16:10:32.267210+0000","last_fullsized":"2024-03-04T16:10:32.267210+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.788235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.788235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265716+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:10:32.265716+0000","last_peered":"2024-03-04T16:10:32.265716+0000","last_clean":"2024-03-04T16:10:32.265716+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:10:32.265716+0000","last_undegraded":"2024-03-04T16:10:32.265716+0000","last_fullsized":"2024-03-04T16:10:32.265716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":331,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.280434+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:10:32.280434+0000","last_peered":"2024-03-04T16:10:32.280434+0000","last_clean":"2024-03-04T16:10:32.280434+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:10:32.280434+0000","last_undegraded":"2024-03-04T16:10:32.280434+0000","last_fullsized":"2024-03-04T16:10:32.280434+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:29.792605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275105+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:10:32.275105+0000","last_peered":"2024-03-04T16:10:32.275105+0000","last_clean":"2024-03-04T16:10:32.275105+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:10:32.275105+0000","last_undegraded":"2024-03-04T16:10:32.275105+0000","last_fullsized":"2024-03-04T16:10:32.275105+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.303288+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288153+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:10:32.288153+0000","last_peered":"2024-03-04T16:10:32.288153+0000","last_clean":"2024-03-04T16:10:32.288153+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:10:32.288153+0000","last_undegraded":"2024-03-04T16:10:32.288153+0000","last_fullsized":"2024-03-04T16:10:32.288153+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.269427+0000","last_change":"2024-03-04T16:09:13.858427+0000","last_active":"2024-03-04T16:10:32.269427+0000","last_peered":"2024-03-04T16:10:32.269427+0000","last_clean":"2024-03-04T16:10:32.269427+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:10:32.269427+0000","last_undegraded":"2024-03-04T16:10:32.269427+0000","last_fullsized":"2024-03-04T16:10:32.269427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.858316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.858316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830273+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":354,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274872+0000","last_change":"2024-03-04T16:10:29.383816+0000","last_active":"2024-03-04T16:10:32.274872+0000","last_peered":"2024-03-04T16:10:32.274872+0000","last_clean":"2024-03-04T16:10:32.274872+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:10:32.274872+0000","last_undegraded":"2024-03-04T16:10:32.274872+0000","last_fullsized":"2024-03-04T16:10:32.274872+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.280165+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:10:32.280165+0000","last_peered":"2024-03-04T16:10:32.280165+0000","last_clean":"2024-03-04T16:10:32.280165+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:10:32.280165+0000","last_undegraded":"2024-03-04T16:10:32.280165+0000","last_fullsized":"2024-03-04T16:10:32.280165+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274678+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:10:32.274678+0000","last_peered":"2024-03-04T16:10:32.274678+0000","last_clean":"2024-03-04T16:10:32.274678+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:10:32.274678+0000","last_undegraded":"2024-03-04T16:10:32.274678+0000","last_fullsized":"2024-03-04T16:10:32.274678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264838+0000","last_change":"2024-03-04T16:10:29.247074+0000","last_active":"2024-03-04T16:10:32.264838+0000","last_peered":"2024-03-04T16:10:32.264838+0000","last_clean":"2024-03-04T16:10:32.264838+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:10:32.264838+0000","last_undegraded":"2024-03-04T16:10:32.264838+0000","last_fullsized":"2024-03-04T16:10:32.264838+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.796905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.769825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":339,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266395+0000","last_change":"2024-03-04T16:10:29.372889+0000","last_active":"2024-03-04T16:10:32.266395+0000","last_peered":"2024-03-04T16:10:32.266395+0000","last_clean":"2024-03-04T16:10:32.266395+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:10:32.266395+0000","last_undegraded":"2024-03-04T16:10:32.266395+0000","last_fullsized":"2024-03-04T16:10:32.266395+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":167,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.271499+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:10:32.271499+0000","last_peered":"2024-03-04T16:10:32.271499+0000","last_clean":"2024-03-04T16:10:32.271499+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:10:32.271499+0000","last_undegraded":"2024-03-04T16:10:32.271499+0000","last_fullsized":"2024-03-04T16:10:32.271499+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:41.311717+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":413,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267842+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:10:32.267842+0000","last_peered":"2024-03-04T16:10:32.267842+0000","last_clean":"2024-03-04T16:10:32.267842+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:10:32.267842+0000","last_undegraded":"2024-03-04T16:10:32.267842+0000","last_fullsized":"2024-03-04T16:10:32.267842+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":332,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267562+0000","last_change":"2024-03-04T16:10:29.311924+0000","last_active":"2024-03-04T16:10:32.267562+0000","last_peered":"2024-03-04T16:10:32.267562+0000","last_clean":"2024-03-04T16:10:32.267562+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:10:32.267562+0000","last_undegraded":"2024-03-04T16:10:32.267562+0000","last_fullsized":"2024-03-04T16:10:32.267562+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:18.820782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.820782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830499+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":337,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.298067+0000","last_change":"2024-03-04T16:10:17.514015+0000","last_active":"2024-03-04T16:10:32.298067+0000","last_peered":"2024-03-04T16:10:32.298067+0000","last_clean":"2024-03-04T16:10:32.298067+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:10:32.298067+0000","last_undegraded":"2024-03-04T16:10:32.298067+0000","last_fullsized":"2024-03-04T16:10:32.298067+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:39.843019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265568+0000","last_change":"2024-03-04T16:10:17.464338+0000","last_active":"2024-03-04T16:10:32.265568+0000","last_peered":"2024-03-04T16:10:32.265568+0000","last_clean":"2024-03-04T16:10:32.265568+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:10:32.265568+0000","last_undegraded":"2024-03-04T16:10:32.265568+0000","last_fullsized":"2024-03-04T16:10:32.265568+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":395,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.298364+0000","last_change":"2024-03-04T16:10:17.498945+0000","last_active":"2024-03-04T16:10:32.298364+0000","last_peered":"2024-03-04T16:10:32.298364+0000","last_clean":"2024-03-04T16:10:32.298364+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:10:32.298364+0000","last_undegraded":"2024-03-04T16:10:32.298364+0000","last_fullsized":"2024-03-04T16:10:32.298364+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":191,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268150+0000","last_change":"2024-03-04T16:08:57.778070+0000","last_active":"2024-03-04T16:10:32.268150+0000","last_peered":"2024-03-04T16:10:32.268150+0000","last_clean":"2024-03-04T16:10:32.268150+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:10:32.268150+0000","last_undegraded":"2024-03-04T16:10:32.268150+0000","last_fullsized":"2024-03-04T16:10:32.268150+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:57.777958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.777958+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267971+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:10:32.267971+0000","last_peered":"2024-03-04T16:10:32.267971+0000","last_clean":"2024-03-04T16:10:32.267971+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:10:32.267971+0000","last_undegraded":"2024-03-04T16:10:32.267971+0000","last_fullsized":"2024-03-04T16:10:32.267971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":26,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265161+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:10:32.265161+0000","last_peered":"2024-03-04T16:10:32.265161+0000","last_clean":"2024-03-04T16:10:32.265161+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:10:32.265161+0000","last_undegraded":"2024-03-04T16:10:32.265161+0000","last_fullsized":"2024-03-04T16:10:32.265161+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:26.382940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":290,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.293327+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:10:32.293327+0000","last_peered":"2024-03-04T16:10:32.293327+0000","last_clean":"2024-03-04T16:10:32.293327+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:10:32.293327+0000","last_undegraded":"2024-03-04T16:10:32.293327+0000","last_fullsized":"2024-03-04T16:10:32.293327+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:05.618945+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266988+0000","last_change":"2024-03-04T16:10:29.243085+0000","last_active":"2024-03-04T16:10:32.266988+0000","last_peered":"2024-03-04T16:10:32.266988+0000","last_clean":"2024-03-04T16:10:32.266988+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:10:32.266988+0000","last_undegraded":"2024-03-04T16:10:32.266988+0000","last_fullsized":"2024-03-04T16:10:32.266988+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":152,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.287177+0000","last_change":"2024-03-04T16:10:09.625107+0000","last_active":"2024-03-04T16:10:32.287177+0000","last_peered":"2024-03-04T16:10:32.287177+0000","last_clean":"2024-03-04T16:10:32.287177+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:10:32.287177+0000","last_undegraded":"2024-03-04T16:10:32.287177+0000","last_fullsized":"2024-03-04T16:10:32.287177+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:09.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.624847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.940215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.269256+0000","last_change":"2024-03-04T16:10:29.370895+0000","last_active":"2024-03-04T16:10:32.269256+0000","last_peered":"2024-03-04T16:10:32.269256+0000","last_clean":"2024-03-04T16:10:32.269256+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:10:32.269256+0000","last_undegraded":"2024-03-04T16:10:32.269256+0000","last_fullsized":"2024-03-04T16:10:32.269256+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.797282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.220990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":297,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265627+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:10:32.265627+0000","last_peered":"2024-03-04T16:10:32.265627+0000","last_clean":"2024-03-04T16:10:32.265627+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:10:32.265627+0000","last_undegraded":"2024-03-04T16:10:32.265627+0000","last_fullsized":"2024-03-04T16:10:32.265627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":204,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275331+0000","last_change":"2024-03-04T16:10:29.242936+0000","last_active":"2024-03-04T16:10:32.275331+0000","last_peered":"2024-03-04T16:10:32.275331+0000","last_clean":"2024-03-04T16:10:32.275331+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:10:32.275331+0000","last_undegraded":"2024-03-04T16:10:32.275331+0000","last_fullsized":"2024-03-04T16:10:32.275331+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.906439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.906439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.040300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.293547+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:10:32.293547+0000","last_peered":"2024-03-04T16:10:32.293547+0000","last_clean":"2024-03-04T16:10:32.293547+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:10:32.293547+0000","last_undegraded":"2024-03-04T16:10:32.293547+0000","last_fullsized":"2024-03-04T16:10:32.293547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.212687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"198'7231","reported_seq":8377,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.451013+0000","last_change":"2024-03-04T16:10:29.433099+0000","last_active":"2024-03-04T16:10:32.451013+0000","last_peered":"2024-03-04T16:10:32.451013+0000","last_clean":"2024-03-04T16:10:16.448068+0000","last_became_active":"2024-03-04T16:10:29.419921+0000","last_became_peered":"2024-03-04T16:10:29.419921+0000","last_unstale":"2024-03-04T16:10:32.451013+0000","last_undegraded":"2024-03-04T16:10:32.451013+0000","last_fullsized":"2024-03-04T16:10:32.451013+0000","mapping_epoch":194,"log_start":"174'2731","ondisk_log_start":"174'2731","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'5174","last_scrub_stamp":"2024-03-04T16:09:57.354140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.354140+0000","objects_scrubbed":5128,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473890816,"num_objects":7231,"num_object_clones":0,"num_object_copies":21693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6726,"num_objects_unfound":0,"num_objects_dirty":7231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7231,"num_write_kb":462784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":32768000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":7231}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":299,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268580+0000","last_change":"2024-03-04T16:10:29.344188+0000","last_active":"2024-03-04T16:10:32.268580+0000","last_peered":"2024-03-04T16:10:32.268580+0000","last_clean":"2024-03-04T16:10:32.268580+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:10:32.268580+0000","last_undegraded":"2024-03-04T16:10:32.268580+0000","last_fullsized":"2024-03-04T16:10:32.268580+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":342,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268490+0000","last_change":"2024-03-04T16:10:09.721121+0000","last_active":"2024-03-04T16:10:32.268490+0000","last_peered":"2024-03-04T16:10:32.268490+0000","last_clean":"2024-03-04T16:10:32.268490+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:10:32.268490+0000","last_undegraded":"2024-03-04T16:10:32.268490+0000","last_fullsized":"2024-03-04T16:10:32.268490+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:10:09.720996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.720996+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:34.736872+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":348,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267846+0000","last_change":"2024-03-04T16:10:29.370436+0000","last_active":"2024-03-04T16:10:32.267846+0000","last_peered":"2024-03-04T16:10:32.267846+0000","last_clean":"2024-03-04T16:10:32.267846+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:10:32.267846+0000","last_undegraded":"2024-03-04T16:10:32.267846+0000","last_fullsized":"2024-03-04T16:10:32.267846+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":402,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267258+0000","last_change":"2024-03-04T16:10:29.252124+0000","last_active":"2024-03-04T16:10:32.267258+0000","last_peered":"2024-03-04T16:10:32.267258+0000","last_clean":"2024-03-04T16:10:32.267258+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:10:32.267258+0000","last_undegraded":"2024-03-04T16:10:32.267258+0000","last_fullsized":"2024-03-04T16:10:32.267258+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:06.869771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.869771+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830244+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"198'7412","reported_seq":8026,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.447846+0000","last_change":"2024-03-04T16:10:30.375150+0000","last_active":"2024-03-04T16:10:32.447846+0000","last_peered":"2024-03-04T16:10:32.447846+0000","last_clean":"2024-03-04T16:10:28.202623+0000","last_became_active":"2024-03-04T16:10:30.345908+0000","last_became_peered":"2024-03-04T16:10:30.345908+0000","last_unstale":"2024-03-04T16:10:32.447846+0000","last_undegraded":"2024-03-04T16:10:32.447846+0000","last_fullsized":"2024-03-04T16:10:32.447846+0000","mapping_epoch":195,"log_start":"175'3171","ondisk_log_start":"175'3171","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485752832,"num_objects":7412,"num_object_clones":0,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7140,"num_objects_unfound":0,"num_objects_dirty":7412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7412,"num_write_kb":474368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":17563648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":7412}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":278,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288350+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:10:32.288350+0000","last_peered":"2024-03-04T16:10:32.288350+0000","last_clean":"2024-03-04T16:10:32.288350+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:10:32.288350+0000","last_undegraded":"2024-03-04T16:10:32.288350+0000","last_fullsized":"2024-03-04T16:10:32.288350+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.271682+0000","last_change":"2024-03-04T16:09:58.574092+0000","last_active":"2024-03-04T16:10:32.271682+0000","last_peered":"2024-03-04T16:10:32.271682+0000","last_clean":"2024-03-04T16:10:32.271682+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:10:32.271682+0000","last_undegraded":"2024-03-04T16:10:32.271682+0000","last_fullsized":"2024-03-04T16:10:32.271682+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:58.573650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.573650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:13.453763+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":325,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264170+0000","last_change":"2024-03-04T16:09:57.809587+0000","last_active":"2024-03-04T16:10:32.264170+0000","last_peered":"2024-03-04T16:10:32.264170+0000","last_clean":"2024-03-04T16:10:32.264170+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:10:32.264170+0000","last_undegraded":"2024-03-04T16:10:32.264170+0000","last_fullsized":"2024-03-04T16:10:32.264170+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:09:57.809460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.809460+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:00.717182+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":196,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274901+0000","last_change":"2024-03-04T16:10:17.478101+0000","last_active":"2024-03-04T16:10:32.274901+0000","last_peered":"2024-03-04T16:10:32.274901+0000","last_clean":"2024-03-04T16:10:32.274901+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:10:32.274901+0000","last_undegraded":"2024-03-04T16:10:32.274901+0000","last_fullsized":"2024-03-04T16:10:32.274901+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:18.939310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.939310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.697848+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"198'7279","reported_seq":8047,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:33.016358+0000","last_change":"2024-03-04T16:10:30.362512+0000","last_active":"2024-03-04T16:10:33.016358+0000","last_peered":"2024-03-04T16:10:33.016358+0000","last_clean":"2024-03-04T16:10:28.227094+0000","last_became_active":"2024-03-04T16:10:30.325895+0000","last_became_peered":"2024-03-04T16:10:30.325895+0000","last_unstale":"2024-03-04T16:10:33.016358+0000","last_undegraded":"2024-03-04T16:10:33.016358+0000","last_fullsized":"2024-03-04T16:10:33.016358+0000","mapping_epoch":195,"log_start":"175'3280","ondisk_log_start":"175'3280","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3999,"log_dups_size":0,"ondisk_log_size":3999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477036544,"num_objects":7279,"num_object_clones":0,"num_object_copies":21837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6930,"num_objects_unfound":0,"num_objects_dirty":7279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7279,"num_write_kb":465856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":22478848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,6],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":7279}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266839+0000","last_change":"2024-03-04T16:10:29.248916+0000","last_active":"2024-03-04T16:10:32.266839+0000","last_peered":"2024-03-04T16:10:32.266839+0000","last_clean":"2024-03-04T16:10:32.266839+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:10:32.266839+0000","last_undegraded":"2024-03-04T16:10:32.266839+0000","last_fullsized":"2024-03-04T16:10:32.266839+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":26,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.278622+0000","last_change":"2024-03-04T16:10:17.466540+0000","last_active":"2024-03-04T16:10:32.278622+0000","last_peered":"2024-03-04T16:10:32.278622+0000","last_clean":"2024-03-04T16:10:32.278622+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:10:32.278622+0000","last_undegraded":"2024-03-04T16:10:32.278622+0000","last_fullsized":"2024-03-04T16:10:32.278622+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.936788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.936788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:36.600071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":312,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.269683+0000","last_change":"2024-03-04T16:10:29.371684+0000","last_active":"2024-03-04T16:10:32.269683+0000","last_peered":"2024-03-04T16:10:32.269683+0000","last_clean":"2024-03-04T16:10:32.269683+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:10:32.269683+0000","last_undegraded":"2024-03-04T16:10:32.269683+0000","last_fullsized":"2024-03-04T16:10:32.269683+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:57.610425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.610425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:23.981998+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":88,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274569+0000","last_change":"2024-03-04T16:10:29.392036+0000","last_active":"2024-03-04T16:10:32.274569+0000","last_peered":"2024-03-04T16:10:32.274569+0000","last_clean":"2024-03-04T16:10:32.274569+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:10:32.274569+0000","last_undegraded":"2024-03-04T16:10:32.274569+0000","last_fullsized":"2024-03-04T16:10:32.274569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"198'7220","reported_seq":7799,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.807350+0000","last_change":"2024-03-04T16:10:30.341634+0000","last_active":"2024-03-04T16:10:32.807350+0000","last_peered":"2024-03-04T16:10:32.807350+0000","last_clean":"2024-03-04T16:10:28.211141+0000","last_became_active":"2024-03-04T16:10:30.305750+0000","last_became_peered":"2024-03-04T16:10:30.305750+0000","last_unstale":"2024-03-04T16:10:32.807350+0000","last_undegraded":"2024-03-04T16:10:32.807350+0000","last_fullsized":"2024-03-04T16:10:32.807350+0000","mapping_epoch":195,"log_start":"177'3603","ondisk_log_start":"177'3603","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473169920,"num_objects":7220,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6972,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7220,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":16318464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":7220}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":173,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266768+0000","last_change":"2024-03-04T16:10:17.462481+0000","last_active":"2024-03-04T16:10:32.266768+0000","last_peered":"2024-03-04T16:10:32.266768+0000","last_clean":"2024-03-04T16:10:32.266768+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:10:32.266768+0000","last_undegraded":"2024-03-04T16:10:32.266768+0000","last_fullsized":"2024-03-04T16:10:32.266768+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.838866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.838866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830363+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":26,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264041+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:10:32.264041+0000","last_peered":"2024-03-04T16:10:32.264041+0000","last_clean":"2024-03-04T16:10:32.264041+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:10:32.264041+0000","last_undegraded":"2024-03-04T16:10:32.264041+0000","last_fullsized":"2024-03-04T16:10:32.264041+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.179174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":311,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267626+0000","last_change":"2024-03-04T16:10:29.354903+0000","last_active":"2024-03-04T16:10:32.267626+0000","last_peered":"2024-03-04T16:10:32.267626+0000","last_clean":"2024-03-04T16:10:32.267626+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:10:32.267626+0000","last_undegraded":"2024-03-04T16:10:32.267626+0000","last_fullsized":"2024-03-04T16:10:32.267626+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":152,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.276208+0000","last_change":"2024-03-04T16:10:12.706114+0000","last_active":"2024-03-04T16:10:32.276208+0000","last_peered":"2024-03-04T16:10:32.276208+0000","last_clean":"2024-03-04T16:10:32.276208+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:10:32.276208+0000","last_undegraded":"2024-03-04T16:10:32.276208+0000","last_fullsized":"2024-03-04T16:10:32.276208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:10:12.705971+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:10:12.705971+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.363548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"198'7331","reported_seq":7402,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.437905+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:10:32.437905+0000","last_peered":"2024-03-04T16:10:32.437905+0000","last_clean":"2024-03-04T16:10:32.437905+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:10:32.437905+0000","last_undegraded":"2024-03-04T16:10:32.437905+0000","last_fullsized":"2024-03-04T16:10:32.437905+0000","mapping_epoch":167,"log_start":"174'2763","ondisk_log_start":"174'2763","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480444416,"num_objects":7331,"num_object_clones":0,"num_object_copies":21993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7331,"num_write_kb":469184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":286,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288842+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:10:32.288842+0000","last_peered":"2024-03-04T16:10:32.288842+0000","last_clean":"2024-03-04T16:10:32.288842+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:10:32.288842+0000","last_undegraded":"2024-03-04T16:10:32.288842+0000","last_fullsized":"2024-03-04T16:10:32.288842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:08.488398+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":326,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268393+0000","last_change":"2024-03-04T16:10:29.374989+0000","last_active":"2024-03-04T16:10:32.268393+0000","last_peered":"2024-03-04T16:10:32.268393+0000","last_clean":"2024-03-04T16:10:32.268393+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:10:32.268393+0000","last_undegraded":"2024-03-04T16:10:32.268393+0000","last_fullsized":"2024-03-04T16:10:32.268393+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":324,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266216+0000","last_change":"2024-03-04T16:10:29.321793+0000","last_active":"2024-03-04T16:10:32.266216+0000","last_peered":"2024-03-04T16:10:32.266216+0000","last_clean":"2024-03-04T16:10:32.266216+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:10:32.266216+0000","last_undegraded":"2024-03-04T16:10:32.266216+0000","last_fullsized":"2024-03-04T16:10:32.266216+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"195'30","reported_seq":492,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.277242+0000","last_change":"2024-03-04T16:10:29.319330+0000","last_active":"2024-03-04T16:10:32.277242+0000","last_peered":"2024-03-04T16:10:32.277242+0000","last_clean":"2024-03-04T16:10:32.277242+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:10:32.277242+0000","last_undegraded":"2024-03-04T16:10:32.277242+0000","last_fullsized":"2024-03-04T16:10:32.277242+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":134,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.280507+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:10:32.280507+0000","last_peered":"2024-03-04T16:10:32.280507+0000","last_clean":"2024-03-04T16:10:32.280507+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:10:32.280507+0000","last_undegraded":"2024-03-04T16:10:32.280507+0000","last_fullsized":"2024-03-04T16:10:32.280507+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:32.697732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"198'7447","reported_seq":7519,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:33.009884+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:10:33.009884+0000","last_peered":"2024-03-04T16:10:33.009884+0000","last_clean":"2024-03-04T16:10:33.009884+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:10:33.009884+0000","last_undegraded":"2024-03-04T16:10:33.009884+0000","last_fullsized":"2024-03-04T16:10:33.009884+0000","mapping_epoch":167,"log_start":"175'3510","ondisk_log_start":"175'3510","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3937,"log_dups_size":0,"ondisk_log_size":3937,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488046592,"num_objects":7447,"num_object_clones":0,"num_object_copies":22341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7447,"num_write_kb":476608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.269843+0000","last_change":"2024-03-04T16:10:29.372155+0000","last_active":"2024-03-04T16:10:32.269843+0000","last_peered":"2024-03-04T16:10:32.269843+0000","last_clean":"2024-03-04T16:10:32.269843+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:10:32.269843+0000","last_undegraded":"2024-03-04T16:10:32.269843+0000","last_fullsized":"2024-03-04T16:10:32.269843+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.867298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.867298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":318,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.273410+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:10:32.273410+0000","last_peered":"2024-03-04T16:10:32.273410+0000","last_clean":"2024-03-04T16:10:32.273410+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:10:32.273410+0000","last_undegraded":"2024-03-04T16:10:32.273410+0000","last_fullsized":"2024-03-04T16:10:32.273410+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.650282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.280191+0000","last_change":"2024-03-04T16:10:17.480211+0000","last_active":"2024-03-04T16:10:32.280191+0000","last_peered":"2024-03-04T16:10:32.280191+0000","last_clean":"2024-03-04T16:10:32.280191+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:10:32.280191+0000","last_undegraded":"2024-03-04T16:10:32.280191+0000","last_fullsized":"2024-03-04T16:10:32.280191+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":362,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.276559+0000","last_change":"2024-03-04T16:08:26.885868+0000","last_active":"2024-03-04T16:10:32.276559+0000","last_peered":"2024-03-04T16:10:32.276559+0000","last_clean":"2024-03-04T16:10:32.276559+0000","last_became_active":"2024-03-04T16:07:36.224375+0000","last_became_peered":"2024-03-04T16:07:36.224375+0000","last_unstale":"2024-03-04T16:10:32.276559+0000","last_undegraded":"2024-03-04T16:10:32.276559+0000","last_fullsized":"2024-03-04T16:10:32.276559+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":26,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.265857+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:10:32.265857+0000","last_peered":"2024-03-04T16:10:32.265857+0000","last_clean":"2024-03-04T16:10:32.265857+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:10:32.265857+0000","last_undegraded":"2024-03-04T16:10:32.265857+0000","last_fullsized":"2024-03-04T16:10:32.265857+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.442466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"198'7191","reported_seq":7267,"reported_epoch":198,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:10:32.455671+0000","last_change":"2024-03-04T16:10:30.370617+0000","last_active":"2024-03-04T16:10:32.455671+0000","last_peered":"2024-03-04T16:10:32.455671+0000","last_clean":"2024-03-04T16:10:28.192446+0000","last_became_active":"2024-03-04T16:10:30.362411+0000","last_became_peered":"2024-03-04T16:10:30.362411+0000","last_unstale":"2024-03-04T16:10:32.455671+0000","last_undegraded":"2024-03-04T16:10:32.455671+0000","last_fullsized":"2024-03-04T16:10:32.455671+0000","mapping_epoch":195,"log_start":"174'2701","ondisk_log_start":"174'2701","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471269376,"num_objects":7191,"num_object_clones":0,"num_object_copies":21573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7191,"num_objects_unfound":0,"num_objects_dirty":7191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7191,"num_write_kb":460224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":7191}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":293,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266161+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:10:32.266161+0000","last_peered":"2024-03-04T16:10:32.266161+0000","last_clean":"2024-03-04T16:10:32.266161+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:10:32.266161+0000","last_undegraded":"2024-03-04T16:10:32.266161+0000","last_fullsized":"2024-03-04T16:10:32.266161+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":604,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281392+0000","last_change":"2024-03-04T16:10:32.281392+0000","last_active":"2024-03-04T16:10:32.281392+0000","last_peered":"2024-03-04T16:10:32.281392+0000","last_clean":"2024-03-04T16:10:32.281392+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:10:32.281392+0000","last_undegraded":"2024-03-04T16:10:32.281392+0000","last_fullsized":"2024-03-04T16:10:32.281392+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:09:16.827314+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:09:16.827314+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281501+0000","last_change":"2024-03-04T16:10:29.340288+0000","last_active":"2024-03-04T16:10:32.281501+0000","last_peered":"2024-03-04T16:10:32.281501+0000","last_clean":"2024-03-04T16:10:32.281501+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:10:32.281501+0000","last_undegraded":"2024-03-04T16:10:32.281501+0000","last_fullsized":"2024-03-04T16:10:32.281501+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"195'17","reported_seq":80,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266253+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:10:32.266253+0000","last_peered":"2024-03-04T16:10:32.266253+0000","last_clean":"2024-03-04T16:10:32.266253+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:10:32.266253+0000","last_undegraded":"2024-03-04T16:10:32.266253+0000","last_fullsized":"2024-03-04T16:10:32.266253+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:21.436971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268677+0000","last_change":"2024-03-04T16:10:17.465880+0000","last_active":"2024-03-04T16:10:32.268677+0000","last_peered":"2024-03-04T16:10:32.268677+0000","last_clean":"2024-03-04T16:10:32.268677+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:10:32.268677+0000","last_undegraded":"2024-03-04T16:10:32.268677+0000","last_fullsized":"2024-03-04T16:10:32.268677+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.788580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.788580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:17.452556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"198'7287","reported_seq":12826,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.805993+0000","last_change":"2024-03-04T16:10:18.656574+0000","last_active":"2024-03-04T16:10:32.805993+0000","last_peered":"2024-03-04T16:10:32.805993+0000","last_clean":"2024-03-04T16:10:16.411531+0000","last_became_active":"2024-03-04T16:10:18.638347+0000","last_became_peered":"2024-03-04T16:10:18.638347+0000","last_unstale":"2024-03-04T16:10:32.805993+0000","last_undegraded":"2024-03-04T16:10:32.805993+0000","last_fullsized":"2024-03-04T16:10:32.805993+0000","mapping_epoch":191,"log_start":"177'3638","ondisk_log_start":"177'3638","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477560832,"num_objects":7287,"num_object_clones":0,"num_object_copies":21861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4450,"num_objects_unfound":0,"num_objects_dirty":7287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7287,"num_write_kb":466368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2731,"num_bytes_recovered":178978816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":7287}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":294,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.273980+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:10:32.273980+0000","last_peered":"2024-03-04T16:10:32.273980+0000","last_clean":"2024-03-04T16:10:32.273980+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:10:32.273980+0000","last_undegraded":"2024-03-04T16:10:32.273980+0000","last_fullsized":"2024-03-04T16:10:32.273980+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":142,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275020+0000","last_change":"2024-03-04T16:10:11.674758+0000","last_active":"2024-03-04T16:10:32.275020+0000","last_peered":"2024-03-04T16:10:32.275020+0000","last_clean":"2024-03-04T16:10:32.275020+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:10:32.275020+0000","last_undegraded":"2024-03-04T16:10:32.275020+0000","last_fullsized":"2024-03-04T16:10:32.275020+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.674483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.674483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.790473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":781,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266891+0000","last_change":"2024-03-04T16:09:21.827377+0000","last_active":"2024-03-04T16:10:32.266891+0000","last_peered":"2024-03-04T16:10:32.266891+0000","last_clean":"2024-03-04T16:10:32.266891+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:10:32.266891+0000","last_undegraded":"2024-03-04T16:10:32.266891+0000","last_fullsized":"2024-03-04T16:10:32.266891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.827277+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830460+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":462,"num_read_kb":457,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":176,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.286864+0000","last_change":"2024-03-04T16:10:17.517202+0000","last_active":"2024-03-04T16:10:32.286864+0000","last_peered":"2024-03-04T16:10:32.286864+0000","last_clean":"2024-03-04T16:10:32.286864+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:10:32.286864+0000","last_undegraded":"2024-03-04T16:10:32.286864+0000","last_fullsized":"2024-03-04T16:10:32.286864+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:10:00.540297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.540297+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:28.643232+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267065+0000","last_change":"2024-03-04T16:09:55.100999+0000","last_active":"2024-03-04T16:10:32.267065+0000","last_peered":"2024-03-04T16:10:32.267065+0000","last_clean":"2024-03-04T16:10:32.267065+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:10:32.267065+0000","last_undegraded":"2024-03-04T16:10:32.267065+0000","last_fullsized":"2024-03-04T16:10:32.267065+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.100871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.100871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.457895+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"198'7202","reported_seq":12211,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:33.023818+0000","last_change":"2024-03-04T16:10:18.699609+0000","last_active":"2024-03-04T16:10:33.023818+0000","last_peered":"2024-03-04T16:10:33.023818+0000","last_clean":"2024-03-04T16:10:16.408274+0000","last_became_active":"2024-03-04T16:10:18.621652+0000","last_became_peered":"2024-03-04T16:10:18.621652+0000","last_unstale":"2024-03-04T16:10:33.023818+0000","last_undegraded":"2024-03-04T16:10:33.023818+0000","last_fullsized":"2024-03-04T16:10:33.023818+0000","mapping_epoch":191,"log_start":"175'3275","ondisk_log_start":"175'3275","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3927,"log_dups_size":0,"ondisk_log_size":3927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471990272,"num_objects":7202,"num_object_clones":0,"num_object_copies":21606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4639,"num_objects_unfound":0,"num_objects_dirty":7202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":460928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2464,"num_bytes_recovered":161480704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":7202}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274031+0000","last_change":"2024-03-04T16:10:29.247615+0000","last_active":"2024-03-04T16:10:32.274031+0000","last_peered":"2024-03-04T16:10:32.274031+0000","last_clean":"2024-03-04T16:10:32.274031+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:10:32.274031+0000","last_undegraded":"2024-03-04T16:10:32.274031+0000","last_fullsized":"2024-03-04T16:10:32.274031+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.995029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.995029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.091071+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.277092+0000","last_change":"2024-03-04T16:10:17.491352+0000","last_active":"2024-03-04T16:10:32.277092+0000","last_peered":"2024-03-04T16:10:32.277092+0000","last_clean":"2024-03-04T16:10:32.277092+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:10:32.277092+0000","last_undegraded":"2024-03-04T16:10:32.277092+0000","last_fullsized":"2024-03-04T16:10:32.277092+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":265,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266339+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:10:32.266339+0000","last_peered":"2024-03-04T16:10:32.266339+0000","last_clean":"2024-03-04T16:10:32.266339+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:10:32.266339+0000","last_undegraded":"2024-03-04T16:10:32.266339+0000","last_fullsized":"2024-03-04T16:10:32.266339+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.268500+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:10:32.268500+0000","last_peered":"2024-03-04T16:10:32.268500+0000","last_clean":"2024-03-04T16:10:32.268500+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:10:32.268500+0000","last_undegraded":"2024-03-04T16:10:32.268500+0000","last_fullsized":"2024-03-04T16:10:32.268500+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.179084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"198'7369","reported_seq":7454,"reported_epoch":198,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:10:32.956248+0000","last_change":"2024-03-04T16:10:18.656245+0000","last_active":"2024-03-04T16:10:32.956248+0000","last_peered":"2024-03-04T16:10:32.956248+0000","last_clean":"2024-03-04T16:10:16.417752+0000","last_became_active":"2024-03-04T16:10:18.594911+0000","last_became_peered":"2024-03-04T16:10:18.594911+0000","last_unstale":"2024-03-04T16:10:32.956248+0000","last_undegraded":"2024-03-04T16:10:32.956248+0000","last_fullsized":"2024-03-04T16:10:32.956248+0000","mapping_epoch":191,"log_start":"175'3384","ondisk_log_start":"175'3384","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3985,"log_dups_size":0,"ondisk_log_size":3985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482934784,"num_objects":7369,"num_object_clones":0,"num_object_copies":22107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7369,"num_objects_unfound":0,"num_objects_dirty":7369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":471616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":7369}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":300,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.275196+0000","last_change":"2024-03-04T16:10:17.467416+0000","last_active":"2024-03-04T16:10:32.275196+0000","last_peered":"2024-03-04T16:10:32.275196+0000","last_clean":"2024-03-04T16:10:32.275196+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:10:32.275196+0000","last_undegraded":"2024-03-04T16:10:32.275196+0000","last_fullsized":"2024-03-04T16:10:32.275196+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.314502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.314502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:40.307403+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":317,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267877+0000","last_change":"2024-03-04T16:10:29.368316+0000","last_active":"2024-03-04T16:10:32.267877+0000","last_peered":"2024-03-04T16:10:32.267877+0000","last_clean":"2024-03-04T16:10:32.267877+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:10:32.267877+0000","last_undegraded":"2024-03-04T16:10:32.267877+0000","last_fullsized":"2024-03-04T16:10:32.267877+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":322,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.274770+0000","last_change":"2024-03-04T16:10:01.014399+0000","last_active":"2024-03-04T16:10:32.274770+0000","last_peered":"2024-03-04T16:10:32.274770+0000","last_clean":"2024-03-04T16:10:32.274770+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:10:32.274770+0000","last_undegraded":"2024-03-04T16:10:32.274770+0000","last_fullsized":"2024-03-04T16:10:32.274770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:10:01.014225+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.014225+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:08.192541+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":48,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266221+0000","last_change":"2024-03-04T16:10:29.369889+0000","last_active":"2024-03-04T16:10:32.266221+0000","last_peered":"2024-03-04T16:10:32.266221+0000","last_clean":"2024-03-04T16:10:32.266221+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:10:32.266221+0000","last_undegraded":"2024-03-04T16:10:32.266221+0000","last_fullsized":"2024-03-04T16:10:32.266221+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:10:03.917637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.917637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:09.946968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"198'7292","reported_seq":11904,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.462753+0000","last_change":"2024-03-04T16:10:29.455466+0000","last_active":"2024-03-04T16:10:32.462753+0000","last_peered":"2024-03-04T16:10:32.462753+0000","last_clean":"2024-03-04T16:10:16.433006+0000","last_became_active":"2024-03-04T16:10:29.431833+0000","last_became_peered":"2024-03-04T16:10:29.431833+0000","last_unstale":"2024-03-04T16:10:32.462753+0000","last_undegraded":"2024-03-04T16:10:32.462753+0000","last_fullsized":"2024-03-04T16:10:32.462753+0000","mapping_epoch":194,"log_start":"174'2804","ondisk_log_start":"174'2804","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477888512,"num_objects":7292,"num_object_clones":0,"num_object_copies":21876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6736,"num_objects_unfound":0,"num_objects_dirty":7292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7292,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2262,"num_bytes_recovered":148242432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,2],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,7","objects":7292}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":306,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267875+0000","last_change":"2024-03-04T16:10:29.245334+0000","last_active":"2024-03-04T16:10:32.267875+0000","last_peered":"2024-03-04T16:10:32.267875+0000","last_clean":"2024-03-04T16:10:32.267875+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:10:32.267875+0000","last_undegraded":"2024-03-04T16:10:32.267875+0000","last_fullsized":"2024-03-04T16:10:32.267875+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:14.851027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:09:14.851027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830563+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":146,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.286978+0000","last_change":"2024-03-04T16:09:24.277958+0000","last_active":"2024-03-04T16:10:32.286978+0000","last_peered":"2024-03-04T16:10:32.286978+0000","last_clean":"2024-03-04T16:10:32.286978+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:10:32.286978+0000","last_undegraded":"2024-03-04T16:10:32.286978+0000","last_fullsized":"2024-03-04T16:10:32.286978+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:24.277674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:24.277674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:49.480623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263984+0000","last_change":"2024-03-04T16:10:29.368068+0000","last_active":"2024-03-04T16:10:32.263984+0000","last_peered":"2024-03-04T16:10:32.263984+0000","last_clean":"2024-03-04T16:10:32.263984+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:10:32.263984+0000","last_undegraded":"2024-03-04T16:10:32.263984+0000","last_fullsized":"2024-03-04T16:10:32.263984+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":392,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.282046+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:10:32.282046+0000","last_peered":"2024-03-04T16:10:32.282046+0000","last_clean":"2024-03-04T16:10:32.282046+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:10:32.282046+0000","last_undegraded":"2024-03-04T16:10:32.282046+0000","last_fullsized":"2024-03-04T16:10:32.282046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:01.984272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"198'7443","reported_seq":7515,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.994570+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:10:32.994570+0000","last_peered":"2024-03-04T16:10:32.994570+0000","last_clean":"2024-03-04T16:10:32.994570+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:10:32.994570+0000","last_undegraded":"2024-03-04T16:10:32.994570+0000","last_fullsized":"2024-03-04T16:10:32.994570+0000","mapping_epoch":167,"log_start":"175'3396","ondisk_log_start":"175'3396","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487784448,"num_objects":7443,"num_object_clones":0,"num_object_copies":22329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7443,"num_write_kb":476352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263739+0000","last_change":"2024-03-04T16:10:29.247041+0000","last_active":"2024-03-04T16:10:32.263739+0000","last_peered":"2024-03-04T16:10:32.263739+0000","last_clean":"2024-03-04T16:10:32.263739+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:10:32.263739+0000","last_undegraded":"2024-03-04T16:10:32.263739+0000","last_fullsized":"2024-03-04T16:10:32.263739+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.293141+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:10:32.293141+0000","last_peered":"2024-03-04T16:10:32.293141+0000","last_clean":"2024-03-04T16:10:32.293141+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:10:32.293141+0000","last_undegraded":"2024-03-04T16:10:32.293141+0000","last_fullsized":"2024-03-04T16:10:32.293141+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.486174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":382,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281933+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:10:32.281933+0000","last_peered":"2024-03-04T16:10:32.281933+0000","last_clean":"2024-03-04T16:10:32.281933+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:10:32.281933+0000","last_undegraded":"2024-03-04T16:10:32.281933+0000","last_fullsized":"2024-03-04T16:10:32.281933+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:43.625140+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263649+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:10:32.263649+0000","last_peered":"2024-03-04T16:10:32.263649+0000","last_clean":"2024-03-04T16:10:32.263649+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:10:32.263649+0000","last_undegraded":"2024-03-04T16:10:32.263649+0000","last_fullsized":"2024-03-04T16:10:32.263649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.310663+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"198'7212","reported_seq":7286,"reported_epoch":198,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:10:32.806953+0000","last_change":"2024-03-04T16:10:30.373285+0000","last_active":"2024-03-04T16:10:32.806953+0000","last_peered":"2024-03-04T16:10:32.806953+0000","last_clean":"2024-03-04T16:10:28.209640+0000","last_became_active":"2024-03-04T16:10:30.368524+0000","last_became_peered":"2024-03-04T16:10:30.368524+0000","last_unstale":"2024-03-04T16:10:32.806953+0000","last_undegraded":"2024-03-04T16:10:32.806953+0000","last_fullsized":"2024-03-04T16:10:32.806953+0000","mapping_epoch":195,"log_start":"177'3578","ondisk_log_start":"177'3578","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3634,"log_dups_size":0,"ondisk_log_size":3634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472645632,"num_objects":7212,"num_object_clones":0,"num_object_copies":21636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7212,"num_objects_unfound":0,"num_objects_dirty":7212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7212,"num_write_kb":461568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":7212}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":305,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.267747+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:10:32.267747+0000","last_peered":"2024-03-04T16:10:32.267747+0000","last_clean":"2024-03-04T16:10:32.267747+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:10:32.267747+0000","last_undegraded":"2024-03-04T16:10:32.267747+0000","last_fullsized":"2024-03-04T16:10:32.267747+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:29.120874+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":392,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264643+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:10:32.264643+0000","last_peered":"2024-03-04T16:10:32.264643+0000","last_clean":"2024-03-04T16:10:32.264643+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:10:32.264643+0000","last_undegraded":"2024-03-04T16:10:32.264643+0000","last_fullsized":"2024-03-04T16:10:32.264643+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.281840+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:10:32.281840+0000","last_peered":"2024-03-04T16:10:32.281840+0000","last_clean":"2024-03-04T16:10:32.281840+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:10:32.281840+0000","last_undegraded":"2024-03-04T16:10:32.281840+0000","last_fullsized":"2024-03-04T16:10:32.281840+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":319,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.276404+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:10:32.276404+0000","last_peered":"2024-03-04T16:10:32.276404+0000","last_clean":"2024-03-04T16:10:32.276404+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:10:32.276404+0000","last_undegraded":"2024-03-04T16:10:32.276404+0000","last_fullsized":"2024-03-04T16:10:32.276404+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"198'7289","reported_seq":7361,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.400656+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:10:32.400656+0000","last_peered":"2024-03-04T16:10:32.400656+0000","last_clean":"2024-03-04T16:10:32.400656+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:10:32.400656+0000","last_undegraded":"2024-03-04T16:10:32.400656+0000","last_fullsized":"2024-03-04T16:10:32.400656+0000","mapping_epoch":167,"log_start":"174'2997","ondisk_log_start":"174'2997","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477691904,"num_objects":7289,"num_object_clones":0,"num_object_copies":21867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":289,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.297957+0000","last_change":"2024-03-04T16:10:29.422572+0000","last_active":"2024-03-04T16:10:32.297957+0000","last_peered":"2024-03-04T16:10:32.297957+0000","last_clean":"2024-03-04T16:10:32.297957+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:10:32.297957+0000","last_undegraded":"2024-03-04T16:10:32.297957+0000","last_fullsized":"2024-03-04T16:10:32.297957+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":303,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266982+0000","last_change":"2024-03-04T16:10:29.344545+0000","last_active":"2024-03-04T16:10:32.266982+0000","last_peered":"2024-03-04T16:10:32.266982+0000","last_clean":"2024-03-04T16:10:32.266982+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:10:32.266982+0000","last_undegraded":"2024-03-04T16:10:32.266982+0000","last_fullsized":"2024-03-04T16:10:32.266982+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:35.954055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":304,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263735+0000","last_change":"2024-03-04T16:10:17.472526+0000","last_active":"2024-03-04T16:10:32.263735+0000","last_peered":"2024-03-04T16:10:32.263735+0000","last_clean":"2024-03-04T16:10:32.263735+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:10:32.263735+0000","last_undegraded":"2024-03-04T16:10:32.263735+0000","last_fullsized":"2024-03-04T16:10:32.263735+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":344,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264546+0000","last_change":"2024-03-04T16:10:29.365556+0000","last_active":"2024-03-04T16:10:32.264546+0000","last_peered":"2024-03-04T16:10:32.264546+0000","last_clean":"2024-03-04T16:10:32.264546+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:10:32.264546+0000","last_undegraded":"2024-03-04T16:10:32.264546+0000","last_fullsized":"2024-03-04T16:10:32.264546+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:10:03.830634+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.830634+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.966501+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"198'7396","reported_seq":11815,"reported_epoch":198,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:10:32.805458+0000","last_change":"2024-03-04T16:10:29.383810+0000","last_active":"2024-03-04T16:10:32.805458+0000","last_peered":"2024-03-04T16:10:32.805458+0000","last_clean":"2024-03-04T16:10:16.409896+0000","last_became_active":"2024-03-04T16:10:18.627715+0000","last_became_peered":"2024-03-04T16:10:18.627715+0000","last_unstale":"2024-03-04T16:10:32.805458+0000","last_undegraded":"2024-03-04T16:10:32.805458+0000","last_fullsized":"2024-03-04T16:10:32.805458+0000","mapping_epoch":191,"log_start":"178'3722","ondisk_log_start":"178'3722","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3674,"log_dups_size":0,"ondisk_log_size":3674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484704256,"num_objects":7396,"num_object_clones":0,"num_object_copies":22188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5123,"num_objects_unfound":0,"num_objects_dirty":7396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7396,"num_write_kb":473344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2169,"num_bytes_recovered":142147584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,2],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1,2","objects":7396}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":32,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.292979+0000","last_change":"2024-03-04T16:10:29.323919+0000","last_active":"2024-03-04T16:10:32.292979+0000","last_peered":"2024-03-04T16:10:32.292979+0000","last_clean":"2024-03-04T16:10:32.292979+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:10:32.292979+0000","last_undegraded":"2024-03-04T16:10:32.292979+0000","last_fullsized":"2024-03-04T16:10:32.292979+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":298,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.277344+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:10:32.277344+0000","last_peered":"2024-03-04T16:10:32.277344+0000","last_clean":"2024-03-04T16:10:32.277344+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:10:32.277344+0000","last_undegraded":"2024-03-04T16:10:32.277344+0000","last_fullsized":"2024-03-04T16:10:32.277344+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.263890+0000","last_change":"2024-03-04T16:10:29.322011+0000","last_active":"2024-03-04T16:10:32.263890+0000","last_peered":"2024-03-04T16:10:32.263890+0000","last_clean":"2024-03-04T16:10:32.263890+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:10:32.263890+0000","last_undegraded":"2024-03-04T16:10:32.263890+0000","last_fullsized":"2024-03-04T16:10:32.263890+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":378,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.266009+0000","last_change":"2024-03-04T16:10:29.356897+0000","last_active":"2024-03-04T16:10:32.266009+0000","last_peered":"2024-03-04T16:10:32.266009+0000","last_clean":"2024-03-04T16:10:32.266009+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:10:32.266009+0000","last_undegraded":"2024-03-04T16:10:32.266009+0000","last_fullsized":"2024-03-04T16:10:32.266009+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:09:56.769841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.769841+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.709756+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"198'7146","reported_seq":7218,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.997288+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:10:32.997288+0000","last_peered":"2024-03-04T16:10:32.997288+0000","last_clean":"2024-03-04T16:10:32.997288+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:10:32.997288+0000","last_undegraded":"2024-03-04T16:10:32.997288+0000","last_fullsized":"2024-03-04T16:10:32.997288+0000","mapping_epoch":167,"log_start":"175'3232","ondisk_log_start":"175'3232","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468320256,"num_objects":7146,"num_object_clones":0,"num_object_copies":21438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":457344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":292,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264650+0000","last_change":"2024-03-04T16:10:17.475035+0000","last_active":"2024-03-04T16:10:32.264650+0000","last_peered":"2024-03-04T16:10:32.264650+0000","last_clean":"2024-03-04T16:10:32.264650+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:10:32.264650+0000","last_undegraded":"2024-03-04T16:10:32.264650+0000","last_fullsized":"2024-03-04T16:10:32.264650+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":113,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.269579+0000","last_change":"2024-03-04T16:10:17.496635+0000","last_active":"2024-03-04T16:10:32.269579+0000","last_peered":"2024-03-04T16:10:32.269579+0000","last_clean":"2024-03-04T16:10:32.269579+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:10:32.269579+0000","last_undegraded":"2024-03-04T16:10:32.269579+0000","last_fullsized":"2024-03-04T16:10:32.269579+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'9","last_scrub_stamp":"2024-03-04T16:08:28.762324+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.762324+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:10:36.830201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":310,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.288059+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:10:32.288059+0000","last_peered":"2024-03-04T16:10:32.288059+0000","last_clean":"2024-03-04T16:10:32.288059+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:10:32.288059+0000","last_undegraded":"2024-03-04T16:10:32.288059+0000","last_fullsized":"2024-03-04T16:10:32.288059+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:27.796430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":327,"reported_epoch":198,"state":"active+clean","last_fresh":"2024-03-04T16:10:32.264980+0000","last_change":"2024-03-04T16:10:29.336876+0000","last_active":"2024-03-04T16:10:32.264980+0000","last_peered":"2024-03-04T16:10:32.264980+0000","last_clean":"2024-03-04T16:10:32.264980+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:10:32.264980+0000","last_undegraded":"2024-03-04T16:10:32.264980+0000","last_fullsized":"2024-03-04T16:10:32.264980+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7651131392,"num_objects":116747,"num_object_clones":0,"num_object_copies":350241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":70488,"num_objects_unfound":0,"num_objects_dirty":116747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":116747,"num_write_kb":7471808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10987,"num_bytes_recovered":720044032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23693230080,"data_stored":23693230080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64962,"ondisk_log_size":64962,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":462,"num_read_kb":457,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070562,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2560260,"kb_used_data":2340280,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":91184892,"statfs":{"total":95995035648,"available":93373329408,"internally_reserved":0,"allocated":2396446720,"data_stored":2396027349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":225245642},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364902,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3571856,"kb_used_data":3269960,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":90173296,"statfs":{"total":95995035648,"available":92337455104,"internally_reserved":0,"allocated":3348439040,"data_stored":3348002374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3086040,"kb_used_data":2829964,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90659112,"statfs":{"total":95995035648,"available":92834930688,"internally_reserved":0,"allocated":2897883136,"data_stored":2897450409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3390552,"kb_used_data":3048392,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":90354600,"statfs":{"total":95995035648,"available":92523110400,"internally_reserved":0,"allocated":3121553408,"data_stored":3121123922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":350353864},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953584,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2685300,"kb_used_data":2509736,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91059852,"statfs":{"total":95995035648,"available":93245288448,"internally_reserved":0,"allocated":2569969664,"data_stored":2569564550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,0,0,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626540,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4125820,"kb_used_data":3846124,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":89619332,"statfs":{"total":95995035648,"available":91770195968,"internally_reserved":0,"allocated":3938430976,"data_stored":3938028682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":286390726},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2910978048,"data_stored":2910978048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3373006848,"data_stored":3373006848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2868117504,"data_stored":2868117504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3121283072,"data_stored":3121283072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2580021248,"data_stored":2580021248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3942514688,"data_stored":3942514688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2492596224,"data_stored":2492596224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2404712448,"data_stored":2404712448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:10:38.787 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-04T16:10:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:38.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:38.789 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:38.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:38.924 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:38.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:39.026 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:39.058 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total time run: 90.0177 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total writes made: 121632 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Write size: 65536 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object size: 65536 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Bandwidth (MB/sec): 84.4501 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Bandwidth: 37.8487 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max bandwidth (MB/sec): 211.562 2024-03-04T16:10:39.089 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min bandwidth (MB/sec): 0 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average IOPS: 1351 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev IOPS: 605.579 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max IOPS: 3385 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min IOPS: 0 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average Latency(s): 0.0118361 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Latency(s): 0.0349726 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max latency(s): 2.36607 2024-03-04T16:10:39.090 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min latency(s): 0.00119552 2024-03-04T16:10:39.095 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Cleaning up (deleting benchmark objects) 2024-03-04T16:10:39.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:39.196 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:39 smithi038 ceph-mon[29779]: pgmap v340: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 330 B/s rd, 25 MiB/s wr, 401 op/s; 70488/350835 objects misplaced (20.091%); 60 MiB/s, 965 objects/s recovering 2024-03-04T16:10:39.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:39 smithi038 ceph-mon[29779]: from='client.15282 v1:172.21.15.38:0/2144238048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:10:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:39 smithi038 ceph-mon[38113]: pgmap v340: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 330 B/s rd, 25 MiB/s wr, 401 op/s; 70488/350835 objects misplaced (20.091%); 60 MiB/s, 965 objects/s recovering 2024-03-04T16:10:39.286 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:39 smithi038 ceph-mon[38113]: from='client.15282 v1:172.21.15.38:0/2144238048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:10:39.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:39.323 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:39 smithi176 ceph-mon[33619]: pgmap v340: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 330 B/s rd, 25 MiB/s wr, 401 op/s; 70488/350835 objects misplaced (20.091%); 60 MiB/s, 965 objects/s recovering 2024-03-04T16:10:39.323 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:39 smithi176 ceph-mon[33619]: from='client.15282 v1:172.21.15.38:0/2144238048' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:10:39.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:39.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:39.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:39.584 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.624 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:39.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:39.635 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:10:39.635 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:10:39.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:39.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:39.865 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:39.898 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:39.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:40.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:40.036 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:40.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:40.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:40.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:40.294 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:40 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:10:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:40 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:10:40.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:40.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:10:40.434 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:40.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:40.473 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:40 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:10:40.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:40 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:10:40.474 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:40 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:10:40.474 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:40 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:10:40.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:40.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:40.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:40.701 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:40.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:40.835 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:40.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:40.972 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:40.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:41.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:41.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:41.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:41.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:41.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:41.349 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:41.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:41.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[29779]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 55 MiB/s wr, 885 op/s; 68763/364614 objects misplaced (18.859%); 80 MiB/s, 1.30k objects/s recovering 2024-03-04T16:10:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:10:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:10:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[38113]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 55 MiB/s wr, 885 op/s; 68763/364614 objects misplaced (18.859%); 80 MiB/s, 1.30k objects/s recovering 2024-03-04T16:10:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:10:41.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:41 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:10:41.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:41.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:41 smithi176 ceph-mon[33619]: pgmap v341: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 895 B/s rd, 55 MiB/s wr, 885 op/s; 68763/364614 objects misplaced (18.859%); 80 MiB/s, 1.30k objects/s recovering 2024-03-04T16:10:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:41 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:10:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:41 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:10:41.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.657 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:41.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:41.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:41.793 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:41.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:41.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:41.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:42.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:42.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:42.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.208 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:42.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:42.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:42.340 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:42.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:42.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:42.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:42 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:10:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:42 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:10:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:42 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:10:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:42 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:10:42.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:42 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:10:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:42 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:10:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.625 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:42.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:42.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:42.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:42.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:42.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:42.890 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:42.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:42.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:43.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:43.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:43.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:43.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:43.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:43.437 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:43 smithi038 ceph-mon[29779]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 780 B/s rd, 48 MiB/s wr, 771 op/s; 68763/364614 objects misplaced (18.859%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:10:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:43 smithi038 ceph-mon[38113]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 780 B/s rd, 48 MiB/s wr, 771 op/s; 68763/364614 objects misplaced (18.859%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:10:43.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:43.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:43.581 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:43 smithi176 ceph-mon[33619]: pgmap v342: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 780 B/s rd, 48 MiB/s wr, 771 op/s; 68763/364614 objects misplaced (18.859%); 70 MiB/s, 1.13k objects/s recovering 2024-03-04T16:10:43.682 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.706 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:43.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:10:43.788 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:10:43.788 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-04T16:10:43.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:10:43.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:43.878 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:43.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:43.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:44.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:44.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:44.198 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:44.300 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:44.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:44.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:44.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:44.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:44.600 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:10:44.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:44.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:44.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:44.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.860 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:44.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:44.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:44.995 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:44.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:45.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:45.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:45.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.258 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:45.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:45.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:45 smithi038 ceph-mon[29779]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 774 op/s; 60942/356535 objects misplaced (17.093%); 66 MiB/s, 1.06k objects/s recovering 2024-03-04T16:10:45.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:45 smithi038 ceph-mon[38113]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 774 op/s; 60942/356535 objects misplaced (17.093%); 66 MiB/s, 1.06k objects/s recovering 2024-03-04T16:10:45.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:45 smithi176 ceph-mon[33619]: pgmap v343: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 774 op/s; 60942/356535 objects misplaced (17.093%); 66 MiB/s, 1.06k objects/s recovering 2024-03-04T16:10:45.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:45.649 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:45.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:45.782 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:45.884 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:45.960 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:45.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:46.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:46.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:46.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:46.213 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:46.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:46.353 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:46.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:46.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:46.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:46.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:46 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:46.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:46.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: pgmap v344: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:10:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:10:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:46 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:10:46.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:46.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:46.891 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:46.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:46.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:47.025 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:47.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:47.161 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:47.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:47.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:47.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:47.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:47.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.555 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:47.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:47.688 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:47.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:47.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:47.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:47.947 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:47.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:48.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.085 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:48.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:48.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:48.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:48.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:48.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:48.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:48.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:48.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:48.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:48.602 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:48.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:48.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:48.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:48.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:48.839 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:48 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:10:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:48 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:10:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:48 smithi176 ceph-mon[33619]: pgmap v345: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:48.872 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:48.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:48.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[29779]: pgmap v345: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:10:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:48 smithi038 ceph-mon[38113]: pgmap v345: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.2 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 25 MiB/s wr, 656 op/s; 60942/356535 objects misplaced (17.093%); 56 MiB/s, 898 objects/s recovering 2024-03-04T16:10:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:48.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:49.100 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:49.132 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:49.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:49.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:49.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:49.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:49.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:49.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:49.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:49.658 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:49.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:49.787 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:49.889 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:49.926 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:49.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:50.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:50.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:50.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:50.184 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:50.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:50.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:50.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.329 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:50.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:50.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:50.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:50.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:50.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:50.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:50.747 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:50.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:50 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:10:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:50 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:10:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:50 smithi176 ceph-mon[33619]: pgmap v346: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 955 op/s; 55855/345786 objects misplaced (16.153%); 73 MiB/s, 1.17k objects/s recovering 2024-03-04T16:10:50.884 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:50.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:50.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[29779]: pgmap v346: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 955 op/s; 55855/345786 objects misplaced (16.153%); 73 MiB/s, 1.17k objects/s recovering 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:10:50.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:50 smithi038 ceph-mon[38113]: pgmap v346: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 955 op/s; 55855/345786 objects misplaced (16.153%); 73 MiB/s, 1.17k objects/s recovering 2024-03-04T16:10:51.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:51.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:51.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:51.149 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:51.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:51.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.274 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:51.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:51.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:51.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:51.543 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:51.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:51.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:51.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:51.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:51 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:10:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:51 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:10:51.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:51.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:51.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:51 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:10:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:51 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:10:51.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:51 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:10:51.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:51 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:10:52.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:52.096 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:52.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:52.222 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:52.324 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:52.354 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:52.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:52.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:52.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:52.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:52.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:52.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:52 smithi176 ceph-mon[33619]: pgmap v347: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 1.5 MiB/s wr, 572 op/s; 55855/345786 objects misplaced (16.153%); 50 MiB/s, 807 objects/s recovering 2024-03-04T16:10:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:52 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:52.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:10:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:10:52.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:52.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:52.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:52.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:52 smithi038 ceph-mon[29779]: pgmap v347: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 1.5 MiB/s wr, 572 op/s; 55855/345786 objects misplaced (16.153%); 50 MiB/s, 807 objects/s recovering 2024-03-04T16:10:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:52 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:52 smithi038 ceph-mon[38113]: pgmap v347: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 7.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 938 B/s rd, 1.5 MiB/s wr, 572 op/s; 55855/345786 objects misplaced (16.153%); 50 MiB/s, 807 objects/s recovering 2024-03-04T16:10:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:52 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:10:53.019 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:53.121 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:53.153 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:53.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.279 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:53.380 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:53.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:53.550 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:53.577 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:10:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:10:53.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:53.692 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:53.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:53.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:10:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:10:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:53.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:53.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:53.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:53.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:10:53.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:10:53.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:10:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:10:53.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:10:54.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:54.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:54.190 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:54.224 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:54.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:54.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:54.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:54.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:54.520 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:54.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:54.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:54 smithi038 ceph-mon[29779]: pgmap v348: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.5 MiB/s wr, 903 op/s; 50205/333891 objects misplaced (15.036%); 70 MiB/s, 1.12k objects/s recovering 2024-03-04T16:10:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:54 smithi038 ceph-mon[29779]: osdmap e200: 8 total, 8 up, 6 in 2024-03-04T16:10:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:54 smithi038 ceph-mon[38113]: pgmap v348: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.5 MiB/s wr, 903 op/s; 50205/333891 objects misplaced (15.036%); 70 MiB/s, 1.12k objects/s recovering 2024-03-04T16:10:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:54 smithi038 ceph-mon[38113]: osdmap e200: 8 total, 8 up, 6 in 2024-03-04T16:10:54.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:54.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:54 smithi176 ceph-mon[33619]: pgmap v348: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 1.5 MiB/s wr, 903 op/s; 50205/333891 objects misplaced (15.036%); 70 MiB/s, 1.12k objects/s recovering 2024-03-04T16:10:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:54 smithi176 ceph-mon[33619]: osdmap e200: 8 total, 8 up, 6 in 2024-03-04T16:10:54.889 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:54.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:54.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:55.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:55.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:55.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:55.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:55.315 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.348 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:55.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:55.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:55.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:55.584 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:55.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:55.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:55.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:55.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:55.882 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:55.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:55.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:56.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:56.071 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:10:56.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:56.126 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:10:56.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:56.166 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:10:56.238 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:10:56.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:56.279 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:10:56.285 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:56.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:56.293 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:10:56.322 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:10:56.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:56 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/62325347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:56.344 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:56 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/62325347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:56.361 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:10:56.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:56.407 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:10:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:10:56.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:56.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:56.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:56.596 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:56.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:56 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/62325347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:10:56.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:56.728 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:56.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:56.864 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:56.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:56.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:57.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:57.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:57.140 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:57.223 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:57 smithi038 ceph-mon[38113]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:57.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:57.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:57.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:57.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.425 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:57.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:57 smithi038 ceph-mon[29779]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:57.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.560 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:57 smithi176 ceph-mon[33619]: pgmap v350: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:57.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:57.700 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:57.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:57.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:57.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:57.981 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:57.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:58.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:10:58.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:10:58.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:58.254 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:58.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:58.389 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:58.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:58 smithi038 ceph-mon[29779]: pgmap v351: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:58.438 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:58 smithi038 ceph-mon[38113]: pgmap v351: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:58.490 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:58.524 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:58 smithi176 ceph-mon[33619]: pgmap v351: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 755 op/s; 50205/333891 objects misplaced (15.036%); 44 MiB/s, 711 objects/s recovering 2024-03-04T16:10:58.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.665 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:10:58.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:10:58.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.799 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:10:58.900 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:10:58.929 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:58.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:10:59.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:59.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:59.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:10:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:59.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:10:59.277 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:10:59.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:59.359 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:59.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:10:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:59 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/270838118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:10:59 smithi038 ceph-mon[29779]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/270838118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:10:59 smithi038 ceph-mon[38113]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.505 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:59.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:10:59.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:59 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/270838118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:10:59 smithi176 ceph-mon[33619]: from='client.35159 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:10:59.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:10:59.646 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:10:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:10:59.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:10:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:59.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:10:59.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:10:59.923 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:10:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:10:59.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:00.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:00.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:00.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:00.219 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:00.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:00.357 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:00.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:00.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:00.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: osdmap e201: 8 total, 8 up, 6 in 2024-03-04T16:11:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:00 smithi176 ceph-mon[33619]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 43956/320655 objects misplaced (13.708%); 64 MiB/s, 1.02k objects/s recovering 2024-03-04T16:11:00.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:00.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:11:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: osdmap e201: 8 total, 8 up, 6 in 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[29779]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 43956/320655 objects misplaced (13.708%); 64 MiB/s, 1.02k objects/s recovering 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: from='client.35159 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: osdmap e201: 8 total, 8 up, 6 in 2024-03-04T16:11:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:00 smithi038 ceph-mon[38113]: pgmap v353: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 43956/320655 objects misplaced (13.708%); 64 MiB/s, 1.02k objects/s recovering 2024-03-04T16:11:00.775 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:00.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:00.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:00.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:01.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:01.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:01.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.198 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:01.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:01.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:01.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:01 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:11:01.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:01 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:11:01.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:01 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:11:01.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:01 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:11:01.333 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:01.435 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:01.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:01.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:11:01.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:01.587 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:01 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:11:01.588 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:01 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:11:01.653 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:01.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:01.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:01.817 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 202, 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:11:01.267299+0000", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:10:28.206965+0000", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5799936, 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:11:01.837 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 8, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "jewel", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:11:01.838 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.839 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:11:01.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.841 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "185", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.842 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.843 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.844 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:11:01.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.847 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.848 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.849 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.851 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6500000953674316, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6500000953674316, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.853 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.854 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.855 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.856 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.857 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.75, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.75, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.309999942779541, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.309999942779541, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.858 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.859 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.860 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.861 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.862 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.863 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5099999904632568, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5099999904632568, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.864 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.865 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:11:01.866 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.867 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.868 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.869 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:11:01.870 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.871 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.871 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.871 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.872 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:11:01.873 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.875 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.876 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.877 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 8, 2024-03-04T16:11:01.878 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_1", 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:08:54.571316+0000", 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-04T16:11:01.879 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "171", 2024-03-04T16:11:01.880 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:11:01.881 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:11:01.882 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:11:01.883 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.884 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 195, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.887 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 195, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.889 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:11:01.890 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.891 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 140, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.893 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.894 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.895 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:11:01.896 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.897 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 194, 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:11:01.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.900 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:11:01.901 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 195, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:11:01.902 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.903 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.904 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:11:01.905 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 195, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.906 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.907 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.908 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.908 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.910 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 190, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:11:01.911 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.912 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:11:01.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:11:01.914 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 197, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:11:01.917 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.918 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:09:39.579253+0000", 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.919 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:10:58.547956+0000", 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.920 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:10:52.574784+0000", 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:10:05.807103+0000", 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.921 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:00.585077+0000", 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.922 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:09:35.821396+0000", 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:10:09.851120+0000", 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:09:47.069616+0000", 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [], 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:11:01.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.1", 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.3", 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:11:01.925 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.4", 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: 4, 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.5", 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.926 INFO:teuthology.orchestra.run.smithi038.stdout: 4, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.6", 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.7", 2024-03-04T16:11:01.927 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.9", 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: 6 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.a", 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.d", 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 7, 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.e", 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 4, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "8.f", 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:11:01.931 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:11:01.932 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:11:01.933 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15171.0:140514 8.9 e202/195 8:9d887fdf:::benchmark_data_smithi038_107057_object18879:head v 202'8660, mlcod=202'8658)", 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:01.888053+0000", 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0038814800000000001, 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039355620000000001, 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888053+0000", 2024-03-04T16:11:01.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888053+0000", 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888054+0000", 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5430000000000001e-06 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:01.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888062+0000", 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0250000000000002e-06 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888073+0000", 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0617e-05 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888080+0000", 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4320000000000002e-06 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888153+0000", 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2156999999999994e-05 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.888177+0000", 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4000000000000001e-05 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.937 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15171.0:140517 8.9 e202/195 8:9c544b56:::benchmark_data_smithi038_107057_object18882:head v 202'8661, mlcod=202'8658)", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:01.890171+0000", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00176362, 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018868450000000001, 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890171+0000", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890171+0000", 2024-03-04T16:11:01.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890172+0000", 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6199999999999999e-06 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:01.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890181+0000", 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7309999999999999e-06 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890189+0000", 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2919999999999996e-06 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890196+0000", 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1559999999999998e-06 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.891347+0000", 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011506439999999999 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.891370+0000", 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3003999999999999e-05 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15171.0:140516 8.2 e202/167 8:46a59993:::benchmark_data_smithi038_107057_object18881:head v 202'8889, mlcod=202'8887)", 2024-03-04T16:11:01.941 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:01.890451+0000", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.001483734, 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0016697280000000001, 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890451+0000", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890451+0000", 2024-03-04T16:11:01.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890452+0000", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.612e-06 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890460+0000", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2190000000000004e-06 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890471+0000", 2024-03-04T16:11:01.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.059e-05 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890479+0000", 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7479999999999993e-06 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890751+0000", 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00027207700000000002 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:01.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:01.890771+0000", 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0276e-05 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 3 2024-03-04T16:11:01.945 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:11:01.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:01.979 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:01.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:02.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:02.111 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:02.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:02.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:02.251 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:02.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:02.387 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:02.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:02.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:02.527 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-mon[33619]: osdmap e202: 8 total, 8 up, 6 in 2024-03-04T16:11:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-mon[33619]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 650 B/s rd, 0 B/s wr, 560 op/s; 43956/320655 objects misplaced (13.708%); 35 MiB/s, 553 objects/s recovering 2024-03-04T16:11:02.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3626090258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:11:02.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:02.672 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:02.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[29779]: osdmap e202: 8 total, 8 up, 6 in 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[29779]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 650 B/s rd, 0 B/s wr, 560 op/s; 43956/320655 objects misplaced (13.708%); 35 MiB/s, 553 objects/s recovering 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3626090258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:11:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[38113]: osdmap e202: 8 total, 8 up, 6 in 2024-03-04T16:11:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[38113]: pgmap v355: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 650 B/s rd, 0 B/s wr, 560 op/s; 43956/320655 objects misplaced (13.708%); 35 MiB/s, 553 objects/s recovering 2024-03-04T16:11:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:02 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3626090258' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:11:02.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:02.817 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:02.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:11:02.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:02.973 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:02.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:03.076 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:03.107 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:03.109 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:03.209 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:03.241 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:03.288 INFO:tasks.thrashosds.thrasher:j is {'epoch': 202, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:11:01.267299+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:10:28.206965+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-04T16:08:54.571316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 195, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 195, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 194, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 195, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 195, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 197, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:09:39.579253+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:10:58.547956+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:10:52.574784+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:10:05.807103+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:11:00.585077+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:09:35.821396+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:10:09.851120+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:09:47.069616+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.1', 'osds': [1, 0, 2]}, {'pgid': '8.3', 'osds': [1, 5, 6]}, {'pgid': '8.4', 'osds': [5, 4, 2]}, {'pgid': '8.5', 'osds': [4, 7, 6]}, {'pgid': '8.6', 'osds': [0, 7, 6]}, {'pgid': '8.7', 'osds': [1, 3, 6]}, {'pgid': '8.9', 'osds': [5, 0, 6]}, {'pgid': '8.a', 'osds': [1, 5, 2]}, {'pgid': '8.d', 'osds': [5, 7, 2]}, {'pgid': '8.e', 'osds': [0, 4, 2]}, {'pgid': '8.f', 'osds': [3, 1, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:11:03.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:11:03.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:03.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[29779]: osdmap e203: 8 total, 8 up, 6 in 2024-03-04T16:11:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:03.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[38113]: osdmap e203: 8 total, 8 up, 6 in 2024-03-04T16:11:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:03.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:03.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:03.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:03 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:03 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:03 smithi176 ceph-mon[33619]: osdmap e203: 8 total, 8 up, 6 in 2024-03-04T16:11:03.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:03.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:03.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:03.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:11:03.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:03.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:03.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:03.947 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:03.947 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:03.959 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:03.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:03.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:04.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:04.106 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:04.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.257 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:04.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:04.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:04.383 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:04 smithi038 ceph-mon[29779]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.54k op/s; 37773/306150 objects misplaced (12.338%); 90 MiB/s, 1.44k objects/s recovering 2024-03-04T16:11:04.384 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:04 smithi038 ceph-mon[38113]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.54k op/s; 37773/306150 objects misplaced (12.338%); 90 MiB/s, 1.44k objects/s recovering 2024-03-04T16:11:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.410 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:04.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:04.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:04.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:04 smithi176 ceph-mon[33619]: pgmap v357: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.54k op/s; 37773/306150 objects misplaced (12.338%); 90 MiB/s, 1.44k objects/s recovering 2024-03-04T16:11:04.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:04.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:04.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.844 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:04.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:04.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:04.979 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:04.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:05.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:05.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:05.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:05.257 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:05.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.409 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:05.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:05.510 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:05.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:05.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:05.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:05.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:05.861 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:05.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:05.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:05.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:06.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:06.135 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:06.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:06.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:06.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.422 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:06.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:06.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:06.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:06 smithi176 ceph-mon[33619]: osdmap e204: 8 total, 8 up, 6 in 2024-03-04T16:11:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:06 smithi176 ceph-mon[33619]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 37773/306150 objects misplaced (12.338%); 45 MiB/s, 716 objects/s recovering 2024-03-04T16:11:06.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:06.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:06.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[38113]: osdmap e204: 8 total, 8 up, 6 in 2024-03-04T16:11:06.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[38113]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 37773/306150 objects misplaced (12.338%); 45 MiB/s, 716 objects/s recovering 2024-03-04T16:11:06.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:06.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:06.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[29779]: osdmap e204: 8 total, 8 up, 6 in 2024-03-04T16:11:06.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:06 smithi038 ceph-mon[29779]: pgmap v359: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 806 op/s; 37773/306150 objects misplaced (12.338%); 45 MiB/s, 716 objects/s recovering 2024-03-04T16:11:06.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:06.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:06.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.834 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:06.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:06.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:06.969 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:06.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:07.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:07.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.104 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:07.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:07.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:07.236 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:07.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:07.385 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:07.487 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:07.522 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:07 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:11:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:07 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:11:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:07.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:07.661 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:07.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:07.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:07.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:07.921 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:07.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:07.937 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:08.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:08.083 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:08.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:08.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:08.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:08.364 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:08.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:08 smithi038 ceph-mon[29779]: pgmap v360: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 829 B/s rd, 0 B/s wr, 783 op/s; 37773/306150 objects misplaced (12.338%); 44 MiB/s, 696 objects/s recovering 2024-03-04T16:11:08.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:08 smithi038 ceph-mon[38113]: pgmap v360: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 829 B/s rd, 0 B/s wr, 783 op/s; 37773/306150 objects misplaced (12.338%); 44 MiB/s, 696 objects/s recovering 2024-03-04T16:11:08.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:08.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:08.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:08.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:08.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:08.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:08 smithi176 ceph-mon[33619]: pgmap v360: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 6.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 829 B/s rd, 0 B/s wr, 783 op/s; 37773/306150 objects misplaced (12.338%); 44 MiB/s, 696 objects/s recovering 2024-03-04T16:11:08.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:08.919 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:08.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:08.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:08.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:11:08.999 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:11:08.999 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:11:08.999 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:11:08.999 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:11:09.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:11:09.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:09.055 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:11:09.062 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:09.095 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:11:09.103 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:11:09.103 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:11:09.103 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:11:09.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:11:09.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:11:09.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:11:09.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:11:09.104 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:11:09.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:09.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:09.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:09.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:09.350 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:09.452 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:09.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:09.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:09.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:09.667 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2420869347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:09.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2420869347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:09.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:09.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2420869347' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:09.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:09.917 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:09.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:10.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:10.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:10.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.206 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:10.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:10.294 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:11:10.300 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:11:10.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:10.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:10.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:10.474 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:10.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:10.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:10.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:10.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:10.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:10.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:10.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:10.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:11.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:11.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:11.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:11.181 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:11.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: pgmap v361: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 31682/291675 objects misplaced (10.862%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:11.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: pgmap v361: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 31682/291675 objects misplaced (10.862%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:11.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.322 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:11.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:11.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:11.461 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:11.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:11.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: pgmap v361: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s; 31682/291675 objects misplaced (10.862%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:11 smithi176 ceph-mon[33619]: osdmap e205: 8 total, 8 up, 6 in 2024-03-04T16:11:11.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.741 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:11.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[29779]: osdmap e205: 8 total, 8 up, 6 in 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:11 smithi038 ceph-mon[38113]: osdmap e205: 8 total, 8 up, 6 in 2024-03-04T16:11:11.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:11.873 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:11.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:11.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:12.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:12.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:12.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:12.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:12.285 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:11:12.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:12.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:12.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:12.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[29779]: from='client.15309 v1:172.21.15.38:0/404108478' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[29779]: osdmap e206: 8 total, 8 up, 6 in 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[29779]: pgmap v364: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 836 B/s rd, 0 B/s wr, 788 op/s; 31682/291675 objects misplaced (10.862%); 45 MiB/s, 715 objects/s recovering 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[38113]: from='client.15309 v1:172.21.15.38:0/404108478' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[38113]: osdmap e206: 8 total, 8 up, 6 in 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:11:12.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:12 smithi038 ceph-mon[38113]: pgmap v364: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 836 B/s rd, 0 B/s wr, 788 op/s; 31682/291675 objects misplaced (10.862%); 45 MiB/s, 715 objects/s recovering 2024-03-04T16:11:12.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:12.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:12 smithi176 ceph-mon[33619]: from='client.15309 v1:172.21.15.38:0/404108478' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:12 smithi176 ceph-mon[33619]: osdmap e206: 8 total, 8 up, 6 in 2024-03-04T16:11:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:11:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:12 smithi176 ceph-mon[33619]: pgmap v364: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 8 active+remapped+backfilling, 137 active+clean; 5.9 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 836 B/s rd, 0 B/s wr, 788 op/s; 31682/291675 objects misplaced (10.862%); 45 MiB/s, 715 objects/s recovering 2024-03-04T16:11:12.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:12.737 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:12.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:12.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:12.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:12.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:13.007 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-03-04T16:11:10.209981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6359292278,"num_objects":97225,"num_object_clones":0,"num_object_copies":291675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31682,"num_objects_unfound":0,"num_objects_dirty":97225,"num_whiteouts":0,"num_read":1417,"num_read_kb":1263,"num_write":146854,"num_write_kb":7786910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42195,"num_bytes_recovered":2751409450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65720,"ondisk_log_size":65720,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":18505960,"kb_used_data":16659632,"kb_used_omap":9,"kb_used_meta":1846006,"kb_avail":543964952,"statfs":{"total":575970213888,"available":557020110848,"internally_reserved":0,"allocated":17059463168,"data_stored":17057072762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":1890310823},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,4,6,3,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-633077760,"num_objects":-9660,"num_object_clones":0,"num_object_copies":-28980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12274,"num_objects_unfound":0,"num_objects_dirty":-9660,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9660,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8685,"num_bytes_recovered":569114648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.910100"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":306,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337618+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:11:05.337618+0000","last_peered":"2024-03-04T16:11:05.337618+0000","last_clean":"2024-03-04T16:11:05.337618+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:11:05.337618+0000","last_undegraded":"2024-03-04T16:11:05.337618+0000","last_fullsized":"2024-03-04T16:11:05.337618+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342203+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:11:05.342203+0000","last_peered":"2024-03-04T16:11:05.342203+0000","last_clean":"2024-03-04T16:11:05.342203+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:11:05.342203+0000","last_undegraded":"2024-03-04T16:11:05.342203+0000","last_fullsized":"2024-03-04T16:11:05.342203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":158,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343559+0000","last_change":"2024-03-04T16:10:29.339688+0000","last_active":"2024-03-04T16:11:05.343559+0000","last_peered":"2024-03-04T16:11:05.343559+0000","last_clean":"2024-03-04T16:11:05.343559+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:11:05.343559+0000","last_undegraded":"2024-03-04T16:11:05.343559+0000","last_fullsized":"2024-03-04T16:11:05.343559+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":346,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.333790+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:11:05.333790+0000","last_peered":"2024-03-04T16:11:05.333790+0000","last_clean":"2024-03-04T16:11:05.333790+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:11:05.333790+0000","last_undegraded":"2024-03-04T16:11:05.333790+0000","last_fullsized":"2024-03-04T16:11:05.333790+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.733521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347054+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:11:05.347054+0000","last_peered":"2024-03-04T16:11:05.347054+0000","last_clean":"2024-03-04T16:11:05.347054+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:11:05.347054+0000","last_undegraded":"2024-03-04T16:11:05.347054+0000","last_fullsized":"2024-03-04T16:11:05.347054+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.333894+0000","last_change":"2024-03-04T16:10:29.325452+0000","last_active":"2024-03-04T16:11:05.333894+0000","last_peered":"2024-03-04T16:11:05.333894+0000","last_clean":"2024-03-04T16:11:05.333894+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:11:05.333894+0000","last_undegraded":"2024-03-04T16:11:05.333894+0000","last_fullsized":"2024-03-04T16:11:05.333894+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":315,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346585+0000","last_change":"2024-03-04T16:10:29.251726+0000","last_active":"2024-03-04T16:11:05.346585+0000","last_peered":"2024-03-04T16:11:05.346585+0000","last_clean":"2024-03-04T16:11:05.346585+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:11:05.346585+0000","last_undegraded":"2024-03-04T16:11:05.346585+0000","last_fullsized":"2024-03-04T16:11:05.346585+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":397,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345312+0000","last_change":"2024-03-04T16:10:17.503786+0000","last_active":"2024-03-04T16:11:05.345312+0000","last_peered":"2024-03-04T16:11:05.345312+0000","last_clean":"2024-03-04T16:11:05.345312+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:11:05.345312+0000","last_undegraded":"2024-03-04T16:11:05.345312+0000","last_fullsized":"2024-03-04T16:11:05.345312+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":154,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341095+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:11:05.341095+0000","last_peered":"2024-03-04T16:11:05.341095+0000","last_clean":"2024-03-04T16:11:05.341095+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:11:05.341095+0000","last_undegraded":"2024-03-04T16:11:05.341095+0000","last_fullsized":"2024-03-04T16:11:05.341095+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":301,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337081+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:11:05.337081+0000","last_peered":"2024-03-04T16:11:05.337081+0000","last_clean":"2024-03-04T16:11:05.337081+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:11:05.337081+0000","last_undegraded":"2024-03-04T16:11:05.337081+0000","last_fullsized":"2024-03-04T16:11:05.337081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":323,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337427+0000","last_change":"2024-03-04T16:10:29.368250+0000","last_active":"2024-03-04T16:11:05.337427+0000","last_peered":"2024-03-04T16:11:05.337427+0000","last_clean":"2024-03-04T16:11:05.337427+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:11:05.337427+0000","last_undegraded":"2024-03-04T16:11:05.337427+0000","last_fullsized":"2024-03-04T16:11:05.337427+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":358,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335333+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:11:05.335333+0000","last_peered":"2024-03-04T16:11:05.335333+0000","last_clean":"2024-03-04T16:11:05.335333+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:11:05.335333+0000","last_undegraded":"2024-03-04T16:11:05.335333+0000","last_fullsized":"2024-03-04T16:11:05.335333+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":313,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337658+0000","last_change":"2024-03-04T16:10:02.242059+0000","last_active":"2024-03-04T16:11:05.337658+0000","last_peered":"2024-03-04T16:11:05.337658+0000","last_clean":"2024-03-04T16:11:05.337658+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:11:05.337658+0000","last_undegraded":"2024-03-04T16:11:05.337658+0000","last_fullsized":"2024-03-04T16:11:05.337658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.241922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.241922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.249200+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343051+0000","last_change":"2024-03-04T16:10:08.741643+0000","last_active":"2024-03-04T16:11:05.343051+0000","last_peered":"2024-03-04T16:11:05.343051+0000","last_clean":"2024-03-04T16:11:05.343051+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:11:05.343051+0000","last_undegraded":"2024-03-04T16:11:05.343051+0000","last_fullsized":"2024-03-04T16:11:05.343051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:08.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:08.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:15.259742+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341334+0000","last_change":"2024-03-04T16:10:00.097245+0000","last_active":"2024-03-04T16:11:05.341334+0000","last_peered":"2024-03-04T16:11:05.341334+0000","last_clean":"2024-03-04T16:11:05.341334+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:11:05.341334+0000","last_undegraded":"2024-03-04T16:11:05.341334+0000","last_fullsized":"2024-03-04T16:11:05.341334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.097027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.097027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:09.191697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":342,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335230+0000","last_change":"2024-03-04T16:10:17.490328+0000","last_active":"2024-03-04T16:11:05.335230+0000","last_peered":"2024-03-04T16:11:05.335230+0000","last_clean":"2024-03-04T16:11:05.335230+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:11:05.335230+0000","last_undegraded":"2024-03-04T16:11:05.335230+0000","last_fullsized":"2024-03-04T16:11:05.335230+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":305,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335663+0000","last_change":"2024-03-04T16:10:29.330112+0000","last_active":"2024-03-04T16:11:05.335663+0000","last_peered":"2024-03-04T16:11:05.335663+0000","last_clean":"2024-03-04T16:11:05.335663+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:11:05.335663+0000","last_undegraded":"2024-03-04T16:11:05.335663+0000","last_fullsized":"2024-03-04T16:11:05.335663+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.348783+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:11:05.348783+0000","last_peered":"2024-03-04T16:11:05.348783+0000","last_clean":"2024-03-04T16:11:05.348783+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:11:05.348783+0000","last_undegraded":"2024-03-04T16:11:05.348783+0000","last_fullsized":"2024-03-04T16:11:05.348783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":146,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342713+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:11:05.342713+0000","last_peered":"2024-03-04T16:11:05.342713+0000","last_clean":"2024-03-04T16:11:05.342713+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:11:05.342713+0000","last_undegraded":"2024-03-04T16:11:05.342713+0000","last_fullsized":"2024-03-04T16:11:05.342713+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":98,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344695+0000","last_change":"2024-03-04T16:10:17.515216+0000","last_active":"2024-03-04T16:11:05.344695+0000","last_peered":"2024-03-04T16:11:05.344695+0000","last_clean":"2024-03-04T16:11:05.344695+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:11:05.344695+0000","last_undegraded":"2024-03-04T16:11:05.344695+0000","last_fullsized":"2024-03-04T16:11:05.344695+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:10:01.944401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.944401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:10.499604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":305,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346523+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:11:05.346523+0000","last_peered":"2024-03-04T16:11:05.346523+0000","last_clean":"2024-03-04T16:11:05.346523+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:11:05.346523+0000","last_undegraded":"2024-03-04T16:11:05.346523+0000","last_fullsized":"2024-03-04T16:11:05.346523+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.509351+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338826+0000","last_change":"2024-03-04T16:10:17.459301+0000","last_active":"2024-03-04T16:11:05.338826+0000","last_peered":"2024-03-04T16:11:05.338826+0000","last_clean":"2024-03-04T16:11:05.338826+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:11:05.338826+0000","last_undegraded":"2024-03-04T16:11:05.338826+0000","last_fullsized":"2024-03-04T16:11:05.338826+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.804222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.804222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.878528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335434+0000","last_change":"2024-03-04T16:10:17.481021+0000","last_active":"2024-03-04T16:11:05.335434+0000","last_peered":"2024-03-04T16:11:05.335434+0000","last_clean":"2024-03-04T16:11:05.335434+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:11:05.335434+0000","last_undegraded":"2024-03-04T16:11:05.335434+0000","last_fullsized":"2024-03-04T16:11:05.335434+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":329,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345063+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:11:05.345063+0000","last_peered":"2024-03-04T16:11:05.345063+0000","last_clean":"2024-03-04T16:11:05.345063+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:11:05.345063+0000","last_undegraded":"2024-03-04T16:11:05.345063+0000","last_fullsized":"2024-03-04T16:11:05.345063+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344178+0000","last_change":"2024-03-04T16:10:17.485038+0000","last_active":"2024-03-04T16:11:05.344178+0000","last_peered":"2024-03-04T16:11:05.344178+0000","last_clean":"2024-03-04T16:11:05.344178+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:11:05.344178+0000","last_undegraded":"2024-03-04T16:11:05.344178+0000","last_fullsized":"2024-03-04T16:11:05.344178+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:09:01.792870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.792870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":236,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338067+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:11:05.338067+0000","last_peered":"2024-03-04T16:11:05.338067+0000","last_clean":"2024-03-04T16:11:05.338067+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:11:05.338067+0000","last_undegraded":"2024-03-04T16:11:05.338067+0000","last_fullsized":"2024-03-04T16:11:05.338067+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.314870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342472+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:11:05.342472+0000","last_peered":"2024-03-04T16:11:05.342472+0000","last_clean":"2024-03-04T16:11:05.342472+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:11:05.342472+0000","last_undegraded":"2024-03-04T16:11:05.342472+0000","last_fullsized":"2024-03-04T16:11:05.342472+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":166,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343655+0000","last_change":"2024-03-04T16:09:58.487155+0000","last_active":"2024-03-04T16:11:05.343655+0000","last_peered":"2024-03-04T16:11:05.343655+0000","last_clean":"2024-03-04T16:11:05.343655+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:11:05.343655+0000","last_undegraded":"2024-03-04T16:11:05.343655+0000","last_fullsized":"2024-03-04T16:11:05.343655+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:58.487024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.487024+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:24.715565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":298,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.348933+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:11:05.348933+0000","last_peered":"2024-03-04T16:11:05.348933+0000","last_clean":"2024-03-04T16:11:05.348933+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:11:05.348933+0000","last_undegraded":"2024-03-04T16:11:05.348933+0000","last_fullsized":"2024-03-04T16:11:05.348933+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343893+0000","last_change":"2024-03-04T16:10:17.479793+0000","last_active":"2024-03-04T16:11:05.343893+0000","last_peered":"2024-03-04T16:11:05.343893+0000","last_clean":"2024-03-04T16:11:05.343893+0000","last_became_active":"2024-03-04T16:10:17.478936+0000","last_became_peered":"2024-03-04T16:10:17.478936+0000","last_unstale":"2024-03-04T16:11:05.343893+0000","last_undegraded":"2024-03-04T16:11:05.343893+0000","last_fullsized":"2024-03-04T16:11:05.343893+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344342+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:11:05.344342+0000","last_peered":"2024-03-04T16:11:05.344342+0000","last_clean":"2024-03-04T16:11:05.344342+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:11:05.344342+0000","last_undegraded":"2024-03-04T16:11:05.344342+0000","last_fullsized":"2024-03-04T16:11:05.344342+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":80,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338563+0000","last_change":"2024-03-04T16:10:29.384325+0000","last_active":"2024-03-04T16:11:05.338563+0000","last_peered":"2024-03-04T16:11:05.338563+0000","last_clean":"2024-03-04T16:11:05.338563+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:11:05.338563+0000","last_undegraded":"2024-03-04T16:11:05.338563+0000","last_fullsized":"2024-03-04T16:11:05.338563+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342966+0000","last_change":"2024-03-04T16:10:17.457894+0000","last_active":"2024-03-04T16:11:05.342966+0000","last_peered":"2024-03-04T16:11:05.342966+0000","last_clean":"2024-03-04T16:11:05.342966+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:11:05.342966+0000","last_undegraded":"2024-03-04T16:11:05.342966+0000","last_fullsized":"2024-03-04T16:11:05.342966+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:09:19.779373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.334061+0000","last_change":"2024-03-04T16:10:29.238069+0000","last_active":"2024-03-04T16:11:05.334061+0000","last_peered":"2024-03-04T16:11:05.334061+0000","last_clean":"2024-03-04T16:11:05.334061+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:11:05.334061+0000","last_undegraded":"2024-03-04T16:11:05.334061+0000","last_fullsized":"2024-03-04T16:11:05.334061+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":146,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343467+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:11:05.343467+0000","last_peered":"2024-03-04T16:11:05.343467+0000","last_clean":"2024-03-04T16:11:05.343467+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:11:05.343467+0000","last_undegraded":"2024-03-04T16:11:05.343467+0000","last_fullsized":"2024-03-04T16:11:05.343467+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345525+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:11:05.345525+0000","last_peered":"2024-03-04T16:11:05.345525+0000","last_clean":"2024-03-04T16:11:05.345525+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:11:05.345525+0000","last_undegraded":"2024-03-04T16:11:05.345525+0000","last_fullsized":"2024-03-04T16:11:05.345525+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":308,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347234+0000","last_change":"2024-03-04T16:10:29.250985+0000","last_active":"2024-03-04T16:11:05.347234+0000","last_peered":"2024-03-04T16:11:05.347234+0000","last_clean":"2024-03-04T16:11:05.347234+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:11:05.347234+0000","last_undegraded":"2024-03-04T16:11:05.347234+0000","last_fullsized":"2024-03-04T16:11:05.347234+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347650+0000","last_change":"2024-03-04T16:10:48.950468+0000","last_active":"2024-03-04T16:11:05.347650+0000","last_peered":"2024-03-04T16:11:05.347650+0000","last_clean":"2024-03-04T16:11:05.347650+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:11:05.347650+0000","last_undegraded":"2024-03-04T16:11:05.347650+0000","last_fullsized":"2024-03-04T16:11:05.347650+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:48.950175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:48.950175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:52.721632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338229+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:11:05.338229+0000","last_peered":"2024-03-04T16:11:05.338229+0000","last_clean":"2024-03-04T16:11:05.338229+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:11:05.338229+0000","last_undegraded":"2024-03-04T16:11:05.338229+0000","last_fullsized":"2024-03-04T16:11:05.338229+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":343,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344406+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:11:05.344406+0000","last_peered":"2024-03-04T16:11:05.344406+0000","last_clean":"2024-03-04T16:11:05.344406+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:11:05.344406+0000","last_undegraded":"2024-03-04T16:11:05.344406+0000","last_fullsized":"2024-03-04T16:11:05.344406+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:29.792605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":146,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341201+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:11:05.341201+0000","last_peered":"2024-03-04T16:11:05.341201+0000","last_clean":"2024-03-04T16:11:05.341201+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:11:05.341201+0000","last_undegraded":"2024-03-04T16:11:05.341201+0000","last_fullsized":"2024-03-04T16:11:05.341201+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":306,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346943+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:11:05.346943+0000","last_peered":"2024-03-04T16:11:05.346943+0000","last_clean":"2024-03-04T16:11:05.346943+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:11:05.346943+0000","last_undegraded":"2024-03-04T16:11:05.346943+0000","last_fullsized":"2024-03-04T16:11:05.346943+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347041+0000","last_change":"2024-03-04T16:10:40.920094+0000","last_active":"2024-03-04T16:11:05.347041+0000","last_peered":"2024-03-04T16:11:05.347041+0000","last_clean":"2024-03-04T16:11:05.347041+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:11:05.347041+0000","last_undegraded":"2024-03-04T16:11:05.347041+0000","last_fullsized":"2024-03-04T16:11:05.347041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:40.919972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:40.919972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:47.331325+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":366,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342969+0000","last_change":"2024-03-04T16:10:29.383816+0000","last_active":"2024-03-04T16:11:05.342969+0000","last_peered":"2024-03-04T16:11:05.342969+0000","last_clean":"2024-03-04T16:11:05.342969+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:11:05.342969+0000","last_undegraded":"2024-03-04T16:11:05.342969+0000","last_fullsized":"2024-03-04T16:11:05.342969+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344044+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:11:05.344044+0000","last_peered":"2024-03-04T16:11:05.344044+0000","last_clean":"2024-03-04T16:11:05.344044+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:11:05.344044+0000","last_undegraded":"2024-03-04T16:11:05.344044+0000","last_fullsized":"2024-03-04T16:11:05.344044+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342715+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:11:05.342715+0000","last_peered":"2024-03-04T16:11:05.342715+0000","last_clean":"2024-03-04T16:11:05.342715+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:11:05.342715+0000","last_undegraded":"2024-03-04T16:11:05.342715+0000","last_fullsized":"2024-03-04T16:11:05.342715+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338468+0000","last_change":"2024-03-04T16:10:29.247074+0000","last_active":"2024-03-04T16:11:05.338468+0000","last_peered":"2024-03-04T16:11:05.338468+0000","last_clean":"2024-03-04T16:11:05.338468+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:11:05.338468+0000","last_undegraded":"2024-03-04T16:11:05.338468+0000","last_fullsized":"2024-03-04T16:11:05.338468+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.796905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.769825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":351,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335159+0000","last_change":"2024-03-04T16:10:29.372889+0000","last_active":"2024-03-04T16:11:05.335159+0000","last_peered":"2024-03-04T16:11:05.335159+0000","last_clean":"2024-03-04T16:11:05.335159+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:11:05.335159+0000","last_undegraded":"2024-03-04T16:11:05.335159+0000","last_fullsized":"2024-03-04T16:11:05.335159+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":179,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344677+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:11:05.344677+0000","last_peered":"2024-03-04T16:11:05.344677+0000","last_clean":"2024-03-04T16:11:05.344677+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:11:05.344677+0000","last_undegraded":"2024-03-04T16:11:05.344677+0000","last_fullsized":"2024-03-04T16:11:05.344677+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":432,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337520+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:11:05.337520+0000","last_peered":"2024-03-04T16:11:05.337520+0000","last_clean":"2024-03-04T16:11:05.337520+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:11:05.337520+0000","last_undegraded":"2024-03-04T16:11:05.337520+0000","last_fullsized":"2024-03-04T16:11:05.337520+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":352,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.876897+0000","last_change":"2024-03-04T16:11:05.876940+0000","last_active":"2024-03-04T16:11:05.876897+0000","last_peered":"2024-03-04T16:11:05.876897+0000","last_clean":"2024-03-04T16:11:05.876897+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:11:05.876897+0000","last_undegraded":"2024-03-04T16:11:05.876897+0000","last_fullsized":"2024-03-04T16:11:05.876897+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:11:05.876809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:11:05.876809+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:10.772164+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":349,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345845+0000","last_change":"2024-03-04T16:10:17.514015+0000","last_active":"2024-03-04T16:11:05.345845+0000","last_peered":"2024-03-04T16:11:05.345845+0000","last_clean":"2024-03-04T16:11:05.345845+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:11:05.345845+0000","last_undegraded":"2024-03-04T16:11:05.345845+0000","last_fullsized":"2024-03-04T16:11:05.345845+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338159+0000","last_change":"2024-03-04T16:10:17.464338+0000","last_active":"2024-03-04T16:11:05.338159+0000","last_peered":"2024-03-04T16:11:05.338159+0000","last_clean":"2024-03-04T16:11:05.338159+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:11:05.338159+0000","last_undegraded":"2024-03-04T16:11:05.338159+0000","last_fullsized":"2024-03-04T16:11:05.338159+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":414,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346146+0000","last_change":"2024-03-04T16:10:17.498945+0000","last_active":"2024-03-04T16:11:05.346146+0000","last_peered":"2024-03-04T16:11:05.346146+0000","last_clean":"2024-03-04T16:11:05.346146+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:11:05.346146+0000","last_undegraded":"2024-03-04T16:11:05.346146+0000","last_fullsized":"2024-03-04T16:11:05.346146+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":211,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343135+0000","last_change":"2024-03-04T16:10:50.000077+0000","last_active":"2024-03-04T16:11:05.343135+0000","last_peered":"2024-03-04T16:11:05.343135+0000","last_clean":"2024-03-04T16:11:05.343135+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:11:05.343135+0000","last_undegraded":"2024-03-04T16:11:05.343135+0000","last_fullsized":"2024-03-04T16:11:05.343135+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:10:49.999919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:49.999919+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.077817+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":329,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338539+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:11:05.338539+0000","last_peered":"2024-03-04T16:11:05.338539+0000","last_clean":"2024-03-04T16:11:05.338539+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:11:05.338539+0000","last_undegraded":"2024-03-04T16:11:05.338539+0000","last_fullsized":"2024-03-04T16:11:05.338539+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":38,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338741+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:11:05.338741+0000","last_peered":"2024-03-04T16:11:05.338741+0000","last_clean":"2024-03-04T16:11:05.338741+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:11:05.338741+0000","last_undegraded":"2024-03-04T16:11:05.338741+0000","last_fullsized":"2024-03-04T16:11:05.338741+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:26.382940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346400+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:11:05.346400+0000","last_peered":"2024-03-04T16:11:05.346400+0000","last_clean":"2024-03-04T16:11:05.346400+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:11:05.346400+0000","last_undegraded":"2024-03-04T16:11:05.346400+0000","last_fullsized":"2024-03-04T16:11:05.346400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:05.618945+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342531+0000","last_change":"2024-03-04T16:10:29.243085+0000","last_active":"2024-03-04T16:11:05.342531+0000","last_peered":"2024-03-04T16:11:05.342531+0000","last_clean":"2024-03-04T16:11:05.342531+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:11:05.342531+0000","last_undegraded":"2024-03-04T16:11:05.342531+0000","last_fullsized":"2024-03-04T16:11:05.342531+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":164,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343742+0000","last_change":"2024-03-04T16:10:09.625107+0000","last_active":"2024-03-04T16:11:05.343742+0000","last_peered":"2024-03-04T16:11:05.343742+0000","last_clean":"2024-03-04T16:11:05.343742+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:11:05.343742+0000","last_undegraded":"2024-03-04T16:11:05.343742+0000","last_fullsized":"2024-03-04T16:11:05.343742+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:09.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.624847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.940215+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345809+0000","last_change":"2024-03-04T16:10:29.370895+0000","last_active":"2024-03-04T16:11:05.345809+0000","last_peered":"2024-03-04T16:11:05.345809+0000","last_clean":"2024-03-04T16:11:05.345809+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:11:05.345809+0000","last_undegraded":"2024-03-04T16:11:05.345809+0000","last_fullsized":"2024-03-04T16:11:05.345809+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.797282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.220990+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":309,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337851+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:11:05.337851+0000","last_peered":"2024-03-04T16:11:05.337851+0000","last_clean":"2024-03-04T16:11:05.337851+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:11:05.337851+0000","last_undegraded":"2024-03-04T16:11:05.337851+0000","last_fullsized":"2024-03-04T16:11:05.337851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":216,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341335+0000","last_change":"2024-03-04T16:10:29.242936+0000","last_active":"2024-03-04T16:11:05.341335+0000","last_peered":"2024-03-04T16:11:05.341335+0000","last_clean":"2024-03-04T16:11:05.341335+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:11:05.341335+0000","last_undegraded":"2024-03-04T16:11:05.341335+0000","last_fullsized":"2024-03-04T16:11:05.341335+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.906439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.906439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.040300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346572+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:11:05.346572+0000","last_peered":"2024-03-04T16:11:05.346572+0000","last_clean":"2024-03-04T16:11:05.346572+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:11:05.346572+0000","last_undegraded":"2024-03-04T16:11:05.346572+0000","last_fullsized":"2024-03-04T16:11:05.346572+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.212687+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"204'9037","reported_seq":17463,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:07.492361+0000","last_change":"2024-03-04T16:10:29.433099+0000","last_active":"2024-03-04T16:11:07.492361+0000","last_peered":"2024-03-04T16:11:07.492361+0000","last_clean":"2024-03-04T16:10:16.448068+0000","last_became_active":"2024-03-04T16:10:29.419921+0000","last_became_peered":"2024-03-04T16:10:29.419921+0000","last_unstale":"2024-03-04T16:11:07.492361+0000","last_undegraded":"2024-03-04T16:11:07.492361+0000","last_fullsized":"2024-03-04T16:11:07.492361+0000","mapping_epoch":194,"log_start":"181'4531","ondisk_log_start":"181'4531","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'5174","last_scrub_stamp":"2024-03-04T16:09:57.354140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.354140+0000","objects_scrubbed":5128,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395247616,"num_objects":6031,"num_object_clones":0,"num_object_copies":18093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2425,"num_objects_unfound":0,"num_objects_dirty":6031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9037,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4134,"num_bytes_recovered":270925824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":6031}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337229+0000","last_change":"2024-03-04T16:10:29.344188+0000","last_active":"2024-03-04T16:11:05.337229+0000","last_peered":"2024-03-04T16:11:05.337229+0000","last_clean":"2024-03-04T16:11:05.337229+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:11:05.337229+0000","last_undegraded":"2024-03-04T16:11:05.337229+0000","last_fullsized":"2024-03-04T16:11:05.337229+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":354,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344216+0000","last_change":"2024-03-04T16:10:09.721121+0000","last_active":"2024-03-04T16:11:05.344216+0000","last_peered":"2024-03-04T16:11:05.344216+0000","last_clean":"2024-03-04T16:11:05.344216+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:11:05.344216+0000","last_undegraded":"2024-03-04T16:11:05.344216+0000","last_fullsized":"2024-03-04T16:11:05.344216+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:10:09.720996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.720996+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:34.736872+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":360,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338974+0000","last_change":"2024-03-04T16:10:29.370436+0000","last_active":"2024-03-04T16:11:05.338974+0000","last_peered":"2024-03-04T16:11:05.338974+0000","last_clean":"2024-03-04T16:11:05.338974+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:11:05.338974+0000","last_undegraded":"2024-03-04T16:11:05.338974+0000","last_fullsized":"2024-03-04T16:11:05.338974+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":429,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342791+0000","last_change":"2024-03-04T16:10:39.929934+0000","last_active":"2024-03-04T16:11:05.342791+0000","last_peered":"2024-03-04T16:11:05.342791+0000","last_clean":"2024-03-04T16:11:05.342791+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:11:05.342791+0000","last_undegraded":"2024-03-04T16:11:05.342791+0000","last_fullsized":"2024-03-04T16:11:05.342791+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:10:39.929809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:39.929809+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:43.565691+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"204'9322","reported_seq":19876,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:07.464870+0000","last_change":"2024-03-04T16:10:30.375150+0000","last_active":"2024-03-04T16:11:07.464870+0000","last_peered":"2024-03-04T16:11:07.464870+0000","last_clean":"2024-03-04T16:10:28.202623+0000","last_became_active":"2024-03-04T16:10:30.345908+0000","last_became_peered":"2024-03-04T16:10:30.345908+0000","last_unstale":"2024-03-04T16:11:07.464870+0000","last_undegraded":"2024-03-04T16:11:07.464870+0000","last_fullsized":"2024-03-04T16:11:07.464870+0000","mapping_epoch":195,"log_start":"182'5071","ondisk_log_start":"182'5071","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405012480,"num_objects":6180,"num_object_clones":0,"num_object_copies":18540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1620,"num_objects_unfound":0,"num_objects_dirty":6180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9322,"num_write_kb":496064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5232,"num_bytes_recovered":342884352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":6180}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":290,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347100+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:11:05.347100+0000","last_peered":"2024-03-04T16:11:05.347100+0000","last_clean":"2024-03-04T16:11:05.347100+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:11:05.347100+0000","last_undegraded":"2024-03-04T16:11:05.347100+0000","last_fullsized":"2024-03-04T16:11:05.347100+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344868+0000","last_change":"2024-03-04T16:09:58.574092+0000","last_active":"2024-03-04T16:11:05.344868+0000","last_peered":"2024-03-04T16:11:05.344868+0000","last_clean":"2024-03-04T16:11:05.344868+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:11:05.344868+0000","last_undegraded":"2024-03-04T16:11:05.344868+0000","last_fullsized":"2024-03-04T16:11:05.344868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:58.573650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.573650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:13.453763+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":337,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.334588+0000","last_change":"2024-03-04T16:09:57.809587+0000","last_active":"2024-03-04T16:11:05.334588+0000","last_peered":"2024-03-04T16:11:05.334588+0000","last_clean":"2024-03-04T16:11:05.334588+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:11:05.334588+0000","last_undegraded":"2024-03-04T16:11:05.334588+0000","last_fullsized":"2024-03-04T16:11:05.334588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:09:57.809460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.809460+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":208,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.340994+0000","last_change":"2024-03-04T16:10:17.478101+0000","last_active":"2024-03-04T16:11:05.340994+0000","last_peered":"2024-03-04T16:11:05.340994+0000","last_clean":"2024-03-04T16:11:05.340994+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:11:05.340994+0000","last_undegraded":"2024-03-04T16:11:05.340994+0000","last_fullsized":"2024-03-04T16:11:05.340994+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:18.939310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.939310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"204'9069","reported_seq":16561,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:08.094070+0000","last_change":"2024-03-04T16:10:30.362512+0000","last_active":"2024-03-04T16:11:08.094070+0000","last_peered":"2024-03-04T16:11:08.094070+0000","last_clean":"2024-03-04T16:10:28.227094+0000","last_became_active":"2024-03-04T16:10:30.325895+0000","last_became_peered":"2024-03-04T16:10:30.325895+0000","last_unstale":"2024-03-04T16:11:08.094070+0000","last_undegraded":"2024-03-04T16:11:08.094070+0000","last_fullsized":"2024-03-04T16:11:08.094070+0000","mapping_epoch":195,"log_start":"184'5080","ondisk_log_start":"184'5080","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3989,"log_dups_size":0,"ondisk_log_size":3989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398524416,"num_objects":6081,"num_object_clones":0,"num_object_copies":18243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2867,"num_objects_unfound":0,"num_objects_dirty":6081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9069,"num_write_kb":484800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3699,"num_bytes_recovered":242417664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,6],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":6081}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347109+0000","last_change":"2024-03-04T16:10:29.248916+0000","last_active":"2024-03-04T16:11:05.347109+0000","last_peered":"2024-03-04T16:11:05.347109+0000","last_clean":"2024-03-04T16:11:05.347109+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:11:05.347109+0000","last_undegraded":"2024-03-04T16:11:05.347109+0000","last_fullsized":"2024-03-04T16:11:05.347109+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342987+0000","last_change":"2024-03-04T16:10:17.466540+0000","last_active":"2024-03-04T16:11:05.342987+0000","last_peered":"2024-03-04T16:11:05.342987+0000","last_clean":"2024-03-04T16:11:05.342987+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:11:05.342987+0000","last_undegraded":"2024-03-04T16:11:05.342987+0000","last_fullsized":"2024-03-04T16:11:05.342987+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.936788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.936788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:36.600071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347338+0000","last_change":"2024-03-04T16:10:29.371684+0000","last_active":"2024-03-04T16:11:05.347338+0000","last_peered":"2024-03-04T16:11:05.347338+0000","last_clean":"2024-03-04T16:11:05.347338+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:11:05.347338+0000","last_undegraded":"2024-03-04T16:11:05.347338+0000","last_fullsized":"2024-03-04T16:11:05.347338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:57.610425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.610425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:23.981998+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":100,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342377+0000","last_change":"2024-03-04T16:10:29.392036+0000","last_active":"2024-03-04T16:11:05.342377+0000","last_peered":"2024-03-04T16:11:05.342377+0000","last_clean":"2024-03-04T16:11:05.342377+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:11:05.342377+0000","last_undegraded":"2024-03-04T16:11:05.342377+0000","last_fullsized":"2024-03-04T16:11:05.342377+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"204'9064","reported_seq":17637,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:08.003056+0000","last_change":"2024-03-04T16:10:30.341634+0000","last_active":"2024-03-04T16:11:08.003056+0000","last_peered":"2024-03-04T16:11:08.003056+0000","last_clean":"2024-03-04T16:10:28.211141+0000","last_became_active":"2024-03-04T16:10:30.305750+0000","last_became_peered":"2024-03-04T16:10:30.305750+0000","last_unstale":"2024-03-04T16:11:08.003056+0000","last_undegraded":"2024-03-04T16:11:08.003056+0000","last_fullsized":"2024-03-04T16:11:08.003056+0000","mapping_epoch":195,"log_start":"184'5403","ondisk_log_start":"184'5403","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390856704,"num_objects":5964,"num_object_clones":0,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2278,"num_objects_unfound":0,"num_objects_dirty":5964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9064,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4240,"num_bytes_recovered":277872640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":5964}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":193,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342351+0000","last_change":"2024-03-04T16:10:47.029618+0000","last_active":"2024-03-04T16:11:05.342351+0000","last_peered":"2024-03-04T16:11:05.342351+0000","last_clean":"2024-03-04T16:11:05.342351+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:11:05.342351+0000","last_undegraded":"2024-03-04T16:11:05.342351+0000","last_fullsized":"2024-03-04T16:11:05.342351+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:47.029491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:10:47.029491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:10.136972+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335588+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:11:05.335588+0000","last_peered":"2024-03-04T16:11:05.335588+0000","last_clean":"2024-03-04T16:11:05.335588+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:11:05.335588+0000","last_undegraded":"2024-03-04T16:11:05.335588+0000","last_fullsized":"2024-03-04T16:11:05.335588+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.179174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":323,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341417+0000","last_change":"2024-03-04T16:10:29.354903+0000","last_active":"2024-03-04T16:11:05.341417+0000","last_peered":"2024-03-04T16:11:05.341417+0000","last_clean":"2024-03-04T16:11:05.341417+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:11:05.341417+0000","last_undegraded":"2024-03-04T16:11:05.341417+0000","last_fullsized":"2024-03-04T16:11:05.341417+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":164,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342803+0000","last_change":"2024-03-04T16:10:12.706114+0000","last_active":"2024-03-04T16:11:05.342803+0000","last_peered":"2024-03-04T16:11:05.342803+0000","last_clean":"2024-03-04T16:11:05.342803+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:11:05.342803+0000","last_undegraded":"2024-03-04T16:11:05.342803+0000","last_fullsized":"2024-03-04T16:11:05.342803+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:10:12.705971+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:10:12.705971+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.363548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"204'9196","reported_seq":9279,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:07.485919+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:11:07.485919+0000","last_peered":"2024-03-04T16:11:07.485919+0000","last_clean":"2024-03-04T16:11:07.485919+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:11:07.485919+0000","last_undegraded":"2024-03-04T16:11:07.485919+0000","last_fullsized":"2024-03-04T16:11:07.485919+0000","mapping_epoch":167,"log_start":"181'4663","ondisk_log_start":"181'4663","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4533,"log_dups_size":0,"ondisk_log_size":4533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402915328,"num_objects":6148,"num_object_clones":0,"num_object_copies":18444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9196,"num_write_kb":491008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":298,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347055+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:11:05.347055+0000","last_peered":"2024-03-04T16:11:05.347055+0000","last_clean":"2024-03-04T16:11:05.347055+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:11:05.347055+0000","last_undegraded":"2024-03-04T16:11:05.347055+0000","last_fullsized":"2024-03-04T16:11:05.347055+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:08.488398+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":339,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.336987+0000","last_change":"2024-03-04T16:10:29.374989+0000","last_active":"2024-03-04T16:11:05.336987+0000","last_peered":"2024-03-04T16:11:05.336987+0000","last_clean":"2024-03-04T16:11:05.336987+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:11:05.336987+0000","last_undegraded":"2024-03-04T16:11:05.336987+0000","last_fullsized":"2024-03-04T16:11:05.336987+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":337,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335024+0000","last_change":"2024-03-04T16:10:29.321793+0000","last_active":"2024-03-04T16:11:05.335024+0000","last_peered":"2024-03-04T16:11:05.335024+0000","last_clean":"2024-03-04T16:11:05.335024+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:11:05.335024+0000","last_undegraded":"2024-03-04T16:11:05.335024+0000","last_fullsized":"2024-03-04T16:11:05.335024+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"195'30","reported_seq":512,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344985+0000","last_change":"2024-03-04T16:10:29.319330+0000","last_active":"2024-03-04T16:11:05.344985+0000","last_peered":"2024-03-04T16:11:05.344985+0000","last_clean":"2024-03-04T16:11:05.344985+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:11:05.344985+0000","last_undegraded":"2024-03-04T16:11:05.344985+0000","last_fullsized":"2024-03-04T16:11:05.344985+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":146,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347985+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:11:05.347985+0000","last_peered":"2024-03-04T16:11:05.347985+0000","last_clean":"2024-03-04T16:11:05.347985+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:11:05.347985+0000","last_undegraded":"2024-03-04T16:11:05.347985+0000","last_fullsized":"2024-03-04T16:11:05.347985+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"204'9311","reported_seq":9395,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:08.076279+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:11:08.076279+0000","last_peered":"2024-03-04T16:11:08.076279+0000","last_clean":"2024-03-04T16:11:08.076279+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:11:08.076279+0000","last_undegraded":"2024-03-04T16:11:08.076279+0000","last_fullsized":"2024-03-04T16:11:08.076279+0000","mapping_epoch":167,"log_start":"184'5410","ondisk_log_start":"184'5410","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403111936,"num_objects":6151,"num_object_clones":0,"num_object_copies":18453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9311,"num_write_kb":494784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347442+0000","last_change":"2024-03-04T16:10:44.996119+0000","last_active":"2024-03-04T16:11:05.347442+0000","last_peered":"2024-03-04T16:11:05.347442+0000","last_clean":"2024-03-04T16:11:05.347442+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:11:05.347442+0000","last_undegraded":"2024-03-04T16:11:05.347442+0000","last_fullsized":"2024-03-04T16:11:05.347442+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:44.995993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:44.995993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:07.188647+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343055+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:11:05.343055+0000","last_peered":"2024-03-04T16:11:05.343055+0000","last_clean":"2024-03-04T16:11:05.343055+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:11:05.343055+0000","last_undegraded":"2024-03-04T16:11:05.343055+0000","last_fullsized":"2024-03-04T16:11:05.343055+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:33.650282+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344150+0000","last_change":"2024-03-04T16:10:17.480211+0000","last_active":"2024-03-04T16:11:05.344150+0000","last_peered":"2024-03-04T16:11:05.344150+0000","last_clean":"2024-03-04T16:11:05.344150+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:11:05.344150+0000","last_undegraded":"2024-03-04T16:11:05.344150+0000","last_fullsized":"2024-03-04T16:11:05.344150+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":374,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.350080+0000","last_change":"2024-03-04T16:08:26.885868+0000","last_active":"2024-03-04T16:11:05.350080+0000","last_peered":"2024-03-04T16:11:05.350080+0000","last_clean":"2024-03-04T16:11:05.350080+0000","last_became_active":"2024-03-04T16:07:36.224375+0000","last_became_peered":"2024-03-04T16:07:36.224375+0000","last_unstale":"2024-03-04T16:11:05.350080+0000","last_undegraded":"2024-03-04T16:11:05.350080+0000","last_fullsized":"2024-03-04T16:11:05.350080+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338235+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:11:05.338235+0000","last_peered":"2024-03-04T16:11:05.338235+0000","last_clean":"2024-03-04T16:11:05.338235+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:11:05.338235+0000","last_undegraded":"2024-03-04T16:11:05.338235+0000","last_fullsized":"2024-03-04T16:11:05.338235+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.442466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"204'8963","reported_seq":9051,"reported_epoch":204,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:11:07.435435+0000","last_change":"2024-03-04T16:10:30.370617+0000","last_active":"2024-03-04T16:11:07.435435+0000","last_peered":"2024-03-04T16:11:07.435435+0000","last_clean":"2024-03-04T16:10:28.192446+0000","last_became_active":"2024-03-04T16:10:30.362411+0000","last_became_peered":"2024-03-04T16:10:30.362411+0000","last_unstale":"2024-03-04T16:11:07.435435+0000","last_undegraded":"2024-03-04T16:11:07.435435+0000","last_fullsized":"2024-03-04T16:11:07.435435+0000","mapping_epoch":195,"log_start":"181'4401","ondisk_log_start":"181'4401","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6089,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8963,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":6089}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":305,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338272+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:11:05.338272+0000","last_peered":"2024-03-04T16:11:05.338272+0000","last_clean":"2024-03-04T16:11:05.338272+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:11:05.338272+0000","last_undegraded":"2024-03-04T16:11:05.338272+0000","last_fullsized":"2024-03-04T16:11:05.338272+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":625,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347325+0000","last_change":"2024-03-04T16:10:33.829546+0000","last_active":"2024-03-04T16:11:05.347325+0000","last_peered":"2024-03-04T16:11:05.347325+0000","last_clean":"2024-03-04T16:11:05.347325+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:11:05.347325+0000","last_undegraded":"2024-03-04T16:11:05.347325+0000","last_fullsized":"2024-03-04T16:11:05.347325+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:10:33.829403+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:33.829403+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:36.682319+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344725+0000","last_change":"2024-03-04T16:10:29.340288+0000","last_active":"2024-03-04T16:11:05.344725+0000","last_peered":"2024-03-04T16:11:05.344725+0000","last_clean":"2024-03-04T16:11:05.344725+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:11:05.344725+0000","last_undegraded":"2024-03-04T16:11:05.344725+0000","last_fullsized":"2024-03-04T16:11:05.344725+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"195'17","reported_seq":99,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337420+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:11:05.337420+0000","last_peered":"2024-03-04T16:11:05.337420+0000","last_clean":"2024-03-04T16:11:05.337420+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:11:05.337420+0000","last_undegraded":"2024-03-04T16:11:05.337420+0000","last_fullsized":"2024-03-04T16:11:05.337420+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:21.436971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344275+0000","last_change":"2024-03-04T16:10:17.465880+0000","last_active":"2024-03-04T16:11:05.344275+0000","last_peered":"2024-03-04T16:11:05.344275+0000","last_clean":"2024-03-04T16:11:05.344275+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:11:05.344275+0000","last_undegraded":"2024-03-04T16:11:05.344275+0000","last_fullsized":"2024-03-04T16:11:05.344275+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.788580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.788580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:17.452556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"204'9125","reported_seq":20823,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:07.999560+0000","last_change":"2024-03-04T16:10:18.656574+0000","last_active":"2024-03-04T16:11:07.999560+0000","last_peered":"2024-03-04T16:11:07.999560+0000","last_clean":"2024-03-04T16:10:16.411531+0000","last_became_active":"2024-03-04T16:10:18.638347+0000","last_became_peered":"2024-03-04T16:10:18.638347+0000","last_unstale":"2024-03-04T16:11:07.999560+0000","last_undegraded":"2024-03-04T16:11:07.999560+0000","last_fullsized":"2024-03-04T16:11:07.999560+0000","mapping_epoch":191,"log_start":"184'5438","ondisk_log_start":"184'5438","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3687,"log_dups_size":0,"ondisk_log_size":3687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":18153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1060,"num_objects_unfound":0,"num_objects_dirty":6051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5804,"num_bytes_recovered":380370944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":6051}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":306,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344554+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:11:05.344554+0000","last_peered":"2024-03-04T16:11:05.344554+0000","last_clean":"2024-03-04T16:11:05.344554+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:11:05.344554+0000","last_undegraded":"2024-03-04T16:11:05.344554+0000","last_fullsized":"2024-03-04T16:11:05.344554+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341470+0000","last_change":"2024-03-04T16:10:11.674758+0000","last_active":"2024-03-04T16:11:05.341470+0000","last_peered":"2024-03-04T16:11:05.341470+0000","last_clean":"2024-03-04T16:11:05.341470+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:11:05.341470+0000","last_undegraded":"2024-03-04T16:11:05.341470+0000","last_fullsized":"2024-03-04T16:11:05.341470+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.674483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.674483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.790473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":837,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.710697+0000","last_change":"2024-03-04T16:10:59.956106+0000","last_active":"2024-03-04T16:11:05.710697+0000","last_peered":"2024-03-04T16:11:05.710697+0000","last_clean":"2024-03-04T16:11:05.710697+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:11:05.710697+0000","last_undegraded":"2024-03-04T16:11:05.710697+0000","last_fullsized":"2024-03-04T16:11:05.710697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:10:59.955973+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:10:59.955973+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:20.849731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":497,"num_read_kb":492,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":188,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343463+0000","last_change":"2024-03-04T16:10:17.517202+0000","last_active":"2024-03-04T16:11:05.343463+0000","last_peered":"2024-03-04T16:11:05.343463+0000","last_clean":"2024-03-04T16:11:05.343463+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:11:05.343463+0000","last_undegraded":"2024-03-04T16:11:05.343463+0000","last_fullsized":"2024-03-04T16:11:05.343463+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:10:00.540297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.540297+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:28.643232+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338745+0000","last_change":"2024-03-04T16:09:55.100999+0000","last_active":"2024-03-04T16:11:05.338745+0000","last_peered":"2024-03-04T16:11:05.338745+0000","last_clean":"2024-03-04T16:11:05.338745+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:11:05.338745+0000","last_undegraded":"2024-03-04T16:11:05.338745+0000","last_fullsized":"2024-03-04T16:11:05.338745+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.100871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.100871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.457895+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"204'8980","reported_seq":22255,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:08.097838+0000","last_change":"2024-03-04T16:10:18.699609+0000","last_active":"2024-03-04T16:11:08.097838+0000","last_peered":"2024-03-04T16:11:08.097838+0000","last_clean":"2024-03-04T16:10:16.408274+0000","last_became_active":"2024-03-04T16:10:18.621652+0000","last_became_peered":"2024-03-04T16:10:18.621652+0000","last_unstale":"2024-03-04T16:11:08.097838+0000","last_undegraded":"2024-03-04T16:11:08.097838+0000","last_fullsized":"2024-03-04T16:11:08.097838+0000","mapping_epoch":191,"log_start":"184'5075","ondisk_log_start":"184'5075","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389545984,"num_objects":5944,"num_object_clones":0,"num_object_copies":17832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":243,"num_objects_unfound":0,"num_objects_dirty":5944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6591,"num_bytes_recovered":431947776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":5944}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342306+0000","last_change":"2024-03-04T16:10:29.247615+0000","last_active":"2024-03-04T16:11:05.342306+0000","last_peered":"2024-03-04T16:11:05.342306+0000","last_clean":"2024-03-04T16:11:05.342306+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:11:05.342306+0000","last_undegraded":"2024-03-04T16:11:05.342306+0000","last_fullsized":"2024-03-04T16:11:05.342306+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.995029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.995029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.091071+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.344838+0000","last_change":"2024-03-04T16:10:17.491352+0000","last_active":"2024-03-04T16:11:05.344838+0000","last_peered":"2024-03-04T16:11:05.344838+0000","last_clean":"2024-03-04T16:11:05.344838+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:11:05.344838+0000","last_undegraded":"2024-03-04T16:11:05.344838+0000","last_fullsized":"2024-03-04T16:11:05.344838+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":277,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.336901+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:11:05.336901+0000","last_peered":"2024-03-04T16:11:05.336901+0000","last_clean":"2024-03-04T16:11:05.336901+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:11:05.336901+0000","last_undegraded":"2024-03-04T16:11:05.336901+0000","last_fullsized":"2024-03-04T16:11:05.336901+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337039+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:11:05.337039+0000","last_peered":"2024-03-04T16:11:05.337039+0000","last_clean":"2024-03-04T16:11:05.337039+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:11:05.337039+0000","last_undegraded":"2024-03-04T16:11:05.337039+0000","last_fullsized":"2024-03-04T16:11:05.337039+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:25.179084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"204'9240","reported_seq":9341,"reported_epoch":204,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:11:08.079046+0000","last_change":"2024-03-04T16:10:48.664560+0000","last_active":"2024-03-04T16:11:08.079046+0000","last_peered":"2024-03-04T16:11:08.079046+0000","last_clean":"2024-03-04T16:10:16.417752+0000","last_became_active":"2024-03-04T16:10:18.594911+0000","last_became_peered":"2024-03-04T16:10:18.594911+0000","last_unstale":"2024-03-04T16:11:08.079046+0000","last_undegraded":"2024-03-04T16:11:08.079046+0000","last_fullsized":"2024-03-04T16:11:08.079046+0000","mapping_epoch":191,"log_start":"184'5284","ondisk_log_start":"184'5284","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397017088,"num_objects":6058,"num_object_clones":0,"num_object_copies":18174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6058,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9240,"num_write_kb":489536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2,4","objects":6058}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":313,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.341300+0000","last_change":"2024-03-04T16:10:17.467416+0000","last_active":"2024-03-04T16:11:05.341300+0000","last_peered":"2024-03-04T16:11:05.341300+0000","last_clean":"2024-03-04T16:11:05.341300+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:11:05.341300+0000","last_undegraded":"2024-03-04T16:11:05.341300+0000","last_fullsized":"2024-03-04T16:11:05.341300+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.314502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.314502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":330,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338441+0000","last_change":"2024-03-04T16:10:29.368316+0000","last_active":"2024-03-04T16:11:05.338441+0000","last_peered":"2024-03-04T16:11:05.338441+0000","last_clean":"2024-03-04T16:11:05.338441+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:11:05.338441+0000","last_undegraded":"2024-03-04T16:11:05.338441+0000","last_fullsized":"2024-03-04T16:11:05.338441+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":335,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.342904+0000","last_change":"2024-03-04T16:10:01.014399+0000","last_active":"2024-03-04T16:11:05.342904+0000","last_peered":"2024-03-04T16:11:05.342904+0000","last_clean":"2024-03-04T16:11:05.342904+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:11:05.342904+0000","last_undegraded":"2024-03-04T16:11:05.342904+0000","last_fullsized":"2024-03-04T16:11:05.342904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:10:01.014225+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.014225+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:08.192541+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":67,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.336700+0000","last_change":"2024-03-04T16:10:29.369889+0000","last_active":"2024-03-04T16:11:05.336700+0000","last_peered":"2024-03-04T16:11:05.336700+0000","last_clean":"2024-03-04T16:11:05.336700+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:11:05.336700+0000","last_undegraded":"2024-03-04T16:11:05.336700+0000","last_fullsized":"2024-03-04T16:11:05.336700+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:10:03.917637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.917637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:09.946968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"204'9100","reported_seq":21923,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:07.485412+0000","last_change":"2024-03-04T16:10:29.455466+0000","last_active":"2024-03-04T16:11:07.485412+0000","last_peered":"2024-03-04T16:11:07.485412+0000","last_clean":"2024-03-04T16:10:16.433006+0000","last_became_active":"2024-03-04T16:10:29.431833+0000","last_became_peered":"2024-03-04T16:10:29.431833+0000","last_unstale":"2024-03-04T16:11:07.485412+0000","last_undegraded":"2024-03-04T16:11:07.485412+0000","last_fullsized":"2024-03-04T16:11:07.485412+0000","mapping_epoch":194,"log_start":"181'4604","ondisk_log_start":"181'4604","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4496,"log_dups_size":0,"ondisk_log_size":4496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400228376,"num_objects":6108,"num_object_clones":0,"num_object_copies":18324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2045,"num_objects_unfound":0,"num_objects_dirty":6108,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9100,"num_write_kb":486593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6360,"num_bytes_recovered":416743448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,2],"avail_no_missing":["5","2","6","7"],"object_location_counts":[{"shards":"2,5,7","objects":6108}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343781+0000","last_change":"2024-03-04T16:10:29.245334+0000","last_active":"2024-03-04T16:11:05.343781+0000","last_peered":"2024-03-04T16:11:05.343781+0000","last_clean":"2024-03-04T16:11:05.343781+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:11:05.343781+0000","last_undegraded":"2024-03-04T16:11:05.343781+0000","last_fullsized":"2024-03-04T16:11:05.343781+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:14.851027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:09:14.851027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":158,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.343564+0000","last_change":"2024-03-04T16:09:24.277958+0000","last_active":"2024-03-04T16:11:05.343564+0000","last_peered":"2024-03-04T16:11:05.343564+0000","last_clean":"2024-03-04T16:11:05.343564+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:11:05.343564+0000","last_undegraded":"2024-03-04T16:11:05.343564+0000","last_fullsized":"2024-03-04T16:11:05.343564+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:24.277674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:24.277674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":357,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.334488+0000","last_change":"2024-03-04T16:10:29.368068+0000","last_active":"2024-03-04T16:11:05.334488+0000","last_peered":"2024-03-04T16:11:05.334488+0000","last_clean":"2024-03-04T16:11:05.334488+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:11:05.334488+0000","last_undegraded":"2024-03-04T16:11:05.334488+0000","last_fullsized":"2024-03-04T16:11:05.334488+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":412,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.348685+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:11:05.348685+0000","last_peered":"2024-03-04T16:11:05.348685+0000","last_clean":"2024-03-04T16:11:05.348685+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:11:05.348685+0000","last_undegraded":"2024-03-04T16:11:05.348685+0000","last_fullsized":"2024-03-04T16:11:05.348685+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"204'9408","reported_seq":9492,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:08.061597+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:11:08.061597+0000","last_peered":"2024-03-04T16:11:08.061597+0000","last_clean":"2024-03-04T16:11:08.061597+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:11:08.061597+0000","last_undegraded":"2024-03-04T16:11:08.061597+0000","last_fullsized":"2024-03-04T16:11:08.061597+0000","mapping_epoch":167,"log_start":"184'5396","ondisk_log_start":"184'5396","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400818176,"num_objects":6116,"num_object_clones":0,"num_object_copies":18348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9408,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335147+0000","last_change":"2024-03-04T16:10:29.247041+0000","last_active":"2024-03-04T16:11:05.335147+0000","last_peered":"2024-03-04T16:11:05.335147+0000","last_clean":"2024-03-04T16:11:05.335147+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:11:05.335147+0000","last_undegraded":"2024-03-04T16:11:05.335147+0000","last_fullsized":"2024-03-04T16:11:05.335147+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345693+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:11:05.345693+0000","last_peered":"2024-03-04T16:11:05.345693+0000","last_clean":"2024-03-04T16:11:05.345693+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:11:05.345693+0000","last_undegraded":"2024-03-04T16:11:05.345693+0000","last_fullsized":"2024-03-04T16:11:05.345693+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:12.486174+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":395,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346814+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:11:05.346814+0000","last_peered":"2024-03-04T16:11:05.346814+0000","last_clean":"2024-03-04T16:11:05.346814+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:11:05.346814+0000","last_undegraded":"2024-03-04T16:11:05.346814+0000","last_fullsized":"2024-03-04T16:11:05.346814+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:43.625140+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.334188+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:11:05.334188+0000","last_peered":"2024-03-04T16:11:05.334188+0000","last_clean":"2024-03-04T16:11:05.334188+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:11:05.334188+0000","last_undegraded":"2024-03-04T16:11:05.334188+0000","last_fullsized":"2024-03-04T16:11:05.334188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:20.310663+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"204'9046","reported_seq":9134,"reported_epoch":204,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:11:08.000155+0000","last_change":"2024-03-04T16:10:30.373285+0000","last_active":"2024-03-04T16:11:08.000155+0000","last_peered":"2024-03-04T16:11:08.000155+0000","last_clean":"2024-03-04T16:10:28.209640+0000","last_became_active":"2024-03-04T16:10:30.368524+0000","last_became_peered":"2024-03-04T16:10:30.368524+0000","last_unstale":"2024-03-04T16:11:08.000155+0000","last_undegraded":"2024-03-04T16:11:08.000155+0000","last_fullsized":"2024-03-04T16:11:08.000155+0000","mapping_epoch":195,"log_start":"184'5378","ondisk_log_start":"184'5378","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3668,"log_dups_size":0,"ondisk_log_size":3668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392822784,"num_objects":5994,"num_object_clones":0,"num_object_copies":17982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5994,"num_objects_unfound":0,"num_objects_dirty":5994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9046,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":5994}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337330+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:11:05.337330+0000","last_peered":"2024-03-04T16:11:05.337330+0000","last_clean":"2024-03-04T16:11:05.337330+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:11:05.337330+0000","last_undegraded":"2024-03-04T16:11:05.337330+0000","last_fullsized":"2024-03-04T16:11:05.337330+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:29.120874+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":411,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335744+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:11:05.335744+0000","last_peered":"2024-03-04T16:11:05.335744+0000","last_clean":"2024-03-04T16:11:05.335744+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:11:05.335744+0000","last_undegraded":"2024-03-04T16:11:05.335744+0000","last_fullsized":"2024-03-04T16:11:05.335744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346721+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:11:05.346721+0000","last_peered":"2024-03-04T16:11:05.346721+0000","last_clean":"2024-03-04T16:11:05.346721+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:11:05.346721+0000","last_undegraded":"2024-03-04T16:11:05.346721+0000","last_fullsized":"2024-03-04T16:11:05.346721+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.349920+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:11:05.349920+0000","last_peered":"2024-03-04T16:11:05.349920+0000","last_clean":"2024-03-04T16:11:05.349920+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:11:05.349920+0000","last_undegraded":"2024-03-04T16:11:05.349920+0000","last_fullsized":"2024-03-04T16:11:05.349920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"204'9215","reported_seq":9300,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:07.435302+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:11:07.435302+0000","last_peered":"2024-03-04T16:11:07.435302+0000","last_clean":"2024-03-04T16:11:07.435302+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:11:07.435302+0000","last_undegraded":"2024-03-04T16:11:07.435302+0000","last_fullsized":"2024-03-04T16:11:07.435302+0000","mapping_epoch":167,"log_start":"182'4897","ondisk_log_start":"182'4897","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398655488,"num_objects":6083,"num_object_clones":0,"num_object_copies":18249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":489536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":301,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345741+0000","last_change":"2024-03-04T16:10:29.422572+0000","last_active":"2024-03-04T16:11:05.345741+0000","last_peered":"2024-03-04T16:11:05.345741+0000","last_clean":"2024-03-04T16:11:05.345741+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:11:05.345741+0000","last_undegraded":"2024-03-04T16:11:05.345741+0000","last_fullsized":"2024-03-04T16:11:05.345741+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":315,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338664+0000","last_change":"2024-03-04T16:10:29.344545+0000","last_active":"2024-03-04T16:11:05.338664+0000","last_peered":"2024-03-04T16:11:05.338664+0000","last_clean":"2024-03-04T16:11:05.338664+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:11:05.338664+0000","last_undegraded":"2024-03-04T16:11:05.338664+0000","last_fullsized":"2024-03-04T16:11:05.338664+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.334237+0000","last_change":"2024-03-04T16:10:17.472526+0000","last_active":"2024-03-04T16:11:05.334237+0000","last_peered":"2024-03-04T16:11:05.334237+0000","last_clean":"2024-03-04T16:11:05.334237+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:11:05.334237+0000","last_undegraded":"2024-03-04T16:11:05.334237+0000","last_fullsized":"2024-03-04T16:11:05.334237+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":356,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335573+0000","last_change":"2024-03-04T16:10:29.365556+0000","last_active":"2024-03-04T16:11:05.335573+0000","last_peered":"2024-03-04T16:11:05.335573+0000","last_clean":"2024-03-04T16:11:05.335573+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:11:05.335573+0000","last_undegraded":"2024-03-04T16:11:05.335573+0000","last_fullsized":"2024-03-04T16:11:05.335573+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:10:03.830634+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.830634+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:19.966501+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"204'9221","reported_seq":21054,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:07.996334+0000","last_change":"2024-03-04T16:10:29.383810+0000","last_active":"2024-03-04T16:11:07.996334+0000","last_peered":"2024-03-04T16:11:07.996334+0000","last_clean":"2024-03-04T16:10:16.409896+0000","last_became_active":"2024-03-04T16:10:18.627715+0000","last_became_peered":"2024-03-04T16:10:18.627715+0000","last_unstale":"2024-03-04T16:11:07.996334+0000","last_undegraded":"2024-03-04T16:11:07.996334+0000","last_fullsized":"2024-03-04T16:11:07.996334+0000","mapping_epoch":191,"log_start":"184'5522","ondisk_log_start":"184'5522","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3699,"log_dups_size":0,"ondisk_log_size":3699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403636224,"num_objects":6159,"num_object_clones":0,"num_object_copies":18477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1003,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9221,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5869,"num_bytes_recovered":384630784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,2],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1,2","objects":6159}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345545+0000","last_change":"2024-03-04T16:10:29.323919+0000","last_active":"2024-03-04T16:11:05.345545+0000","last_peered":"2024-03-04T16:11:05.345545+0000","last_clean":"2024-03-04T16:11:05.345545+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:11:05.345545+0000","last_undegraded":"2024-03-04T16:11:05.345545+0000","last_fullsized":"2024-03-04T16:11:05.345545+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.345033+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:11:05.345033+0000","last_peered":"2024-03-04T16:11:05.345033+0000","last_clean":"2024-03-04T16:11:05.345033+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:11:05.345033+0000","last_undegraded":"2024-03-04T16:11:05.345033+0000","last_fullsized":"2024-03-04T16:11:05.345033+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335246+0000","last_change":"2024-03-04T16:10:29.322011+0000","last_active":"2024-03-04T16:11:05.335246+0000","last_peered":"2024-03-04T16:11:05.335246+0000","last_clean":"2024-03-04T16:11:05.335246+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:11:05.335246+0000","last_undegraded":"2024-03-04T16:11:05.335246+0000","last_fullsized":"2024-03-04T16:11:05.335246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":390,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.337244+0000","last_change":"2024-03-04T16:10:29.356897+0000","last_active":"2024-03-04T16:11:05.337244+0000","last_peered":"2024-03-04T16:11:05.337244+0000","last_clean":"2024-03-04T16:11:05.337244+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:11:05.337244+0000","last_undegraded":"2024-03-04T16:11:05.337244+0000","last_fullsized":"2024-03-04T16:11:05.337244+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:09:56.769841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.769841+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:16.709756+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"204'8942","reported_seq":9026,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:08.091691+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:11:08.091691+0000","last_peered":"2024-03-04T16:11:08.091691+0000","last_clean":"2024-03-04T16:11:08.091691+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:11:08.091691+0000","last_undegraded":"2024-03-04T16:11:08.091691+0000","last_fullsized":"2024-03-04T16:11:08.091691+0000","mapping_epoch":167,"log_start":"184'5032","ondisk_log_start":"184'5032","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3910,"log_dups_size":0,"ondisk_log_size":3910,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384696320,"num_objects":5870,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":473984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":304,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.335830+0000","last_change":"2024-03-04T16:10:17.475035+0000","last_active":"2024-03-04T16:11:05.335830+0000","last_peered":"2024-03-04T16:11:05.335830+0000","last_clean":"2024-03-04T16:11:05.335830+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:11:05.335830+0000","last_undegraded":"2024-03-04T16:11:05.335830+0000","last_fullsized":"2024-03-04T16:11:05.335830+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":140,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.347225+0000","last_change":"2024-03-04T16:10:38.935575+0000","last_active":"2024-03-04T16:11:05.347225+0000","last_peered":"2024-03-04T16:11:05.347225+0000","last_clean":"2024-03-04T16:11:05.347225+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:11:05.347225+0000","last_undegraded":"2024-03-04T16:11:05.347225+0000","last_fullsized":"2024-03-04T16:11:05.347225+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:10:38.935466+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:10:38.935466+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.582470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.346858+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:11:05.346858+0000","last_peered":"2024-03-04T16:11:05.346858+0000","last_clean":"2024-03-04T16:11:05.346858+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:11:05.346858+0000","last_undegraded":"2024-03-04T16:11:05.346858+0000","last_fullsized":"2024-03-04T16:11:05.346858+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:27.796430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":339,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-04T16:11:05.338613+0000","last_change":"2024-03-04T16:10:29.336876+0000","last_active":"2024-03-04T16:11:05.338613+0000","last_peered":"2024-03-04T16:11:05.338613+0000","last_clean":"2024-03-04T16:11:05.338613+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:11:05.338613+0000","last_undegraded":"2024-03-04T16:11:05.338613+0000","last_fullsized":"2024-03-04T16:11:05.338613+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6358695960,"num_objects":97027,"num_object_clones":0,"num_object_copies":291081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31682,"num_objects_unfound":0,"num_objects_dirty":97027,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146239,"num_write_kb":7784449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41930,"num_bytes_recovered":2747858968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21502836736,"data_stored":21502820448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65054,"ondisk_log_size":65054,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":497,"num_read_kb":492,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070569,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2190168,"kb_used_data":1978508,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":91554984,"statfs":{"total":95995035648,"available":93752303616,"internally_reserved":0,"allocated":2025992192,"data_stored":2025590753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":216725962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364909,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574983,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3336936,"kb_used_data":2976540,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":90408216,"statfs":{"total":95995035648,"available":92578013184,"internally_reserved":0,"allocated":3047976960,"data_stored":3047558226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2938776,"kb_used_data":2636816,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":90806376,"statfs":{"total":95995035648,"available":92985729024,"internally_reserved":0,"allocated":2700099584,"data_stored":2699694009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149820,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3471380,"kb_used_data":3079244,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":90273772,"statfs":{"total":95995035648,"available":92440342528,"internally_reserved":0,"allocated":3153145856,"data_stored":3152743522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":401537480},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953591,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2793472,"kb_used_data":2555448,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":90951680,"statfs":{"total":95995035648,"available":93134520320,"internally_reserved":0,"allocated":2616778752,"data_stored":2616395642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626547,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3775228,"kb_used_data":3433076,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":89969924,"statfs":{"total":95995035648,"available":92129202176,"internally_reserved":0,"allocated":3515469824,"data_stored":3515090610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":350353862},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2696347648,"data_stored":2696347648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3030716416,"data_stored":3030712344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2383810560,"data_stored":2383806488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3145990144,"data_stored":3145990144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2614820864,"data_stored":2614820864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3511029760,"data_stored":3511025688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2108358656,"data_stored":2108358656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2011762688,"data_stored":2011758616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:11:13.014 INFO:tasks.thrashosds.thrasher:Setting 4.7 to [1, 7, 5, 0, 3, 6] 2024-03-04T16:11:13.014 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.7', '1', '7', '5', '0', '3', '6'] 2024-03-04T16:11:13.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pg-upmap-items 4.7 1 7 5 0 3 6 2024-03-04T16:11:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.016 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:13.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:13.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:13.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:13.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:13.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:13.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:13.277 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:13.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:13.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:13.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:13.544 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:13 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:11:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:13 smithi176 ceph-mon[33619]: osdmap e207: 8 total, 8 up, 6 in 2024-03-04T16:11:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:13.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:13.690 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[29779]: osdmap e207: 8 total, 8 up, 6 in 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4068785955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:11:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[38113]: osdmap e207: 8 total, 8 up, 6 in 2024-03-04T16:11:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:13.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:13.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:13.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:13.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:13.979 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:13.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:13.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:14.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:14.125 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:14.227 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:14.258 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:14.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:14.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.386 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:14.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:14.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:14.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:14 smithi038 ceph-mon[29779]: pgmap v366: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 24831/279513 objects misplaced (8.884%); 104 MiB/s, 1.67k objects/s recovering 2024-03-04T16:11:14.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:14 smithi038 ceph-mon[38113]: pgmap v366: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 24831/279513 objects misplaced (8.884%); 104 MiB/s, 1.67k objects/s recovering 2024-03-04T16:11:14.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:14.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:11:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:14 smithi176 ceph-mon[33619]: pgmap v366: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s; 24831/279513 objects misplaced (8.884%); 104 MiB/s, 1.67k objects/s recovering 2024-03-04T16:11:14.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.700 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:14.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:14.801 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:14.824 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:14.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:14.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:14.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:14.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:15.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:15.099 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:15.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:15.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:15.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:15.366 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:15.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:15.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:15.498 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:15.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:15.638 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:15.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:15.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:15.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:15.895 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:15.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:15.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:15.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:16.025 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:16.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:16.155 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:16.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:16.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:16.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:16.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:16.418 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:16.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:16.553 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:16 smithi176 ceph-mon[33619]: osdmap e208: 8 total, 8 up, 6 in 2024-03-04T16:11:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:16 smithi176 ceph-mon[33619]: pgmap v368: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 772 op/s; 24831/279513 objects misplaced (8.884%); 67 MiB/s, 1.07k objects/s recovering 2024-03-04T16:11:16.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:16.684 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:16 smithi038 ceph-mon[29779]: osdmap e208: 8 total, 8 up, 6 in 2024-03-04T16:11:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:16 smithi038 ceph-mon[29779]: pgmap v368: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 772 op/s; 24831/279513 objects misplaced (8.884%); 67 MiB/s, 1.07k objects/s recovering 2024-03-04T16:11:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:16 smithi038 ceph-mon[38113]: osdmap e208: 8 total, 8 up, 6 in 2024-03-04T16:11:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:16 smithi038 ceph-mon[38113]: pgmap v368: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 772 op/s; 24831/279513 objects misplaced (8.884%); 67 MiB/s, 1.07k objects/s recovering 2024-03-04T16:11:16.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:16.811 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:16.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:16.944 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:16.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:17.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:17.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:17.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:17.197 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:17.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:17.332 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:17.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:17.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:17.584 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:17 smithi176 ceph-mon[33619]: osdmap e209: 8 total, 8 up, 6 in 2024-03-04T16:11:17.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:17 smithi176 ceph-mon[33619]: osdmap e210: 8 total, 8 up, 6 in 2024-03-04T16:11:17.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:17.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:17 smithi038 ceph-mon[29779]: osdmap e209: 8 total, 8 up, 6 in 2024-03-04T16:11:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:17 smithi038 ceph-mon[29779]: osdmap e210: 8 total, 8 up, 6 in 2024-03-04T16:11:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:17 smithi038 ceph-mon[38113]: osdmap e209: 8 total, 8 up, 6 in 2024-03-04T16:11:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:17 smithi038 ceph-mon[38113]: osdmap e210: 8 total, 8 up, 6 in 2024-03-04T16:11:17.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:17.852 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:17.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:17.994 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:17.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:18.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:18.129 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:18.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:18.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.269 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:18.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:18.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.407 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:18.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:18.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:18.549 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:18.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:18.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:18 smithi176 ceph-mon[33619]: pgmap v371: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 986 B/s rd, 0 B/s wr, 782 op/s; 24831/279513 objects misplaced (8.884%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:18.651 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.675 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:18.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:18.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:18 smithi038 ceph-mon[29779]: pgmap v371: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 986 B/s rd, 0 B/s wr, 782 op/s; 24831/279513 objects misplaced (8.884%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:18.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:18 smithi038 ceph-mon[38113]: pgmap v371: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 7 active+remapped+backfilling, 138 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 986 B/s rd, 0 B/s wr, 782 op/s; 24831/279513 objects misplaced (8.884%); 68 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:18.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:18.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:18.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:18.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:18.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:19.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:19.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.068 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:19.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:19.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:19.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:19.347 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:19.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:19.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:19.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:19.608 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:19.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:19.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:19.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:19.862 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:19.964 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:19.993 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:19.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:20.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:20.130 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:20.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:20.264 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:20.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:20.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:20.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:20 smithi176 ceph-mon[33619]: osdmap e211: 8 total, 8 up, 6 in 2024-03-04T16:11:20.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:20 smithi176 ceph-mon[33619]: pgmap v373: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:20.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:20 smithi038 ceph-mon[38113]: osdmap e211: 8 total, 8 up, 6 in 2024-03-04T16:11:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:20 smithi038 ceph-mon[38113]: pgmap v373: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:20 smithi038 ceph-mon[29779]: osdmap e211: 8 total, 8 up, 6 in 2024-03-04T16:11:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:20 smithi038 ceph-mon[29779]: pgmap v373: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:20.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.793 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:20.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:20.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:20.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:20.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:20.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:21.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:21.054 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:21.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:21.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:21.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:21.413 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:21.445 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:21.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:21.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:21.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:21.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:21 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:21 smithi038 ceph-mon[38113]: osdmap e212: 8 total, 8 up, 6 in 2024-03-04T16:11:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:21 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:21.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:21 smithi038 ceph-mon[29779]: osdmap e212: 8 total, 8 up, 6 in 2024-03-04T16:11:21.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:21.839 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:21 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:21 smithi176 ceph-mon[33619]: osdmap e212: 8 total, 8 up, 6 in 2024-03-04T16:11:21.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:21.967 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:21.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:22.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.094 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:22.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:22.219 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:22.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:22.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:22.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:22.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:22.617 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:22.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[38113]: osdmap e213: 8 total, 8 up, 6 in 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[38113]: pgmap v376: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[29779]: osdmap e213: 8 total, 8 up, 6 in 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[29779]: pgmap v376: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:22.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:22.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:22.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:22 smithi176 ceph-mon[33619]: osdmap e213: 8 total, 8 up, 6 in 2024-03-04T16:11:22.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:22 smithi176 ceph-mon[33619]: pgmap v376: 148 pgs: 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 141 active+clean; 5.5 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 729 op/s; 18849/270423 objects misplaced (6.970%); 78 MiB/s, 1.25k objects/s recovering 2024-03-04T16:11:22.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:22.763 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:22.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:22.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:22.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:22.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:23.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:23.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:23.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.152 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:23.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:23.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:23.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:23.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:23.393 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.418 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:23.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:23.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:23.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:11:23.551 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:23.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:23.677 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:23 smithi038 ceph-mon[29779]: osdmap e214: 8 total, 8 up, 6 in 2024-03-04T16:11:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:23.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:23 smithi038 ceph-mon[38113]: osdmap e214: 8 total, 8 up, 6 in 2024-03-04T16:11:23.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:23.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:23.826 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:23 smithi176 ceph-mon[33619]: osdmap e214: 8 total, 8 up, 6 in 2024-03-04T16:11:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:23.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:23.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:23.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:24.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:24.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:24.191 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:24.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:24.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:24.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:24.349 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:11:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:11:24.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:24.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:24.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:24.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:24.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:24 smithi038 ceph-mon[29779]: pgmap v378: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 635 op/s; 13912/262590 objects misplaced (5.298%); 67 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:24.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:24 smithi038 ceph-mon[38113]: pgmap v378: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 635 op/s; 13912/262590 objects misplaced (5.298%); 67 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:24.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:24 smithi176 ceph-mon[33619]: pgmap v378: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 635 op/s; 13912/262590 objects misplaced (5.298%); 67 MiB/s, 1.08k objects/s recovering 2024-03-04T16:11:24.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:24.906 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:24.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:25.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.044 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:25.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:25.145 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:25.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:25.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:25.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:25.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:25.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:25.553 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:25.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:25.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:25.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:25.956 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:25.990 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:25.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:25.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:26.091 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.124 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:26.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:26.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:26.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:26.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:26.395 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:26 smithi038 ceph-mon[29779]: pgmap v379: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:26.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:26 smithi038 ceph-mon[29779]: osdmap e215: 8 total, 8 up, 6 in 2024-03-04T16:11:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:26 smithi038 ceph-mon[38113]: pgmap v379: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:26 smithi038 ceph-mon[38113]: osdmap e215: 8 total, 8 up, 6 in 2024-03-04T16:11:26.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:26.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:26 smithi176 ceph-mon[33619]: pgmap v379: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:26 smithi176 ceph-mon[33619]: osdmap e215: 8 total, 8 up, 6 in 2024-03-04T16:11:26.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:26.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:26.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:26.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:26.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:26.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:26.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:26.905 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:26.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:27.007 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:27.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:27.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:27.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:27.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:27.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:27.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:27.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:27.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.567 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:27.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:27.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:27.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:27.836 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:27.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:27.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:28.079 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:28.111 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:28.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:28.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:28.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:28.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:28.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:28.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:28 smithi176 ceph-mon[33619]: osdmap e216: 8 total, 8 up, 6 in 2024-03-04T16:11:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:28 smithi176 ceph-mon[33619]: pgmap v382: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:28.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:28.664 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:28 smithi038 ceph-mon[29779]: osdmap e216: 8 total, 8 up, 6 in 2024-03-04T16:11:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:28 smithi038 ceph-mon[29779]: pgmap v382: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:28 smithi038 ceph-mon[38113]: osdmap e216: 8 total, 8 up, 6 in 2024-03-04T16:11:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:28 smithi038 ceph-mon[38113]: pgmap v382: 148 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 144 active+clean; 5.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 435 op/s; 13912/262590 objects misplaced (5.298%); 46 MiB/s, 738 objects/s recovering 2024-03-04T16:11:28.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:28.804 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:28.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:28.944 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:28.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:29.045 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:29.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:29.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:29.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:29.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:29.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:29.333 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:29.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:29.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:29.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:29.606 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:29 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:29 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:29 smithi176 ceph-mon[33619]: osdmap e217: 8 total, 8 up, 6 in 2024-03-04T16:11:29.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[29779]: osdmap e217: 8 total, 8 up, 6 in 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:29.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:29 smithi038 ceph-mon[38113]: osdmap e217: 8 total, 8 up, 6 in 2024-03-04T16:11:29.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:29.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:29.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:29.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:29.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:29.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.026 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:30.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:30.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:30.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:30.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:30.303 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:30.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:30.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:30.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:11:30.539 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:30 smithi038 ceph-mon[29779]: pgmap v384: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:30.539 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:30 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:30.539 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:30 smithi038 ceph-mon[38113]: pgmap v384: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:30.540 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:30 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:30.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:30.576 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:30 smithi176 ceph-mon[33619]: pgmap v384: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:30 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:11:30.626 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:30.632 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:30.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:30.710 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:30.717 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:30.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:30.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:30.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:30.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:31.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:31.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.151 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:31.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:31.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.289 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:31.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:31.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:31.428 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:31.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:31.566 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:31.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.701 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:31.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:31.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.839 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:31.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:31.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:31.972 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:31.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:32.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:32.113 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:32.215 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:32.257 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:32.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:32.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:32.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:32.494 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:32.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:32.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:32.664 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:32.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:32.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:32.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:32 smithi038 ceph-mon[38113]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:32.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:32 smithi038 ceph-mon[29779]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:32.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:32 smithi176 ceph-mon[33619]: pgmap v385: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 5.1 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 751 op/s; 11372/249072 objects misplaced (4.566%); 20 MiB/s, 314 objects/s recovering 2024-03-04T16:11:32.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:32.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:32.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:32.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:33.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:33.082 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:33.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:33.219 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:33.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:33.351 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:33.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:33.454 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:33.487 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:33.589 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:33.594 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:33.595 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:33.595 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:33.622 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:33.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:33.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:33.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:33.912 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:33.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:33.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:34.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.049 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:34.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:34.082 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:34.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:34.184 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:34.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:34.317 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:34.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.451 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:34.552 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.590 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:34.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:34.827 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:34 smithi176 ceph-mon[33619]: pgmap v386: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.51k op/s; 9321/229764 objects misplaced (4.057%); 27 MiB/s, 428 objects/s recovering 2024-03-04T16:11:34.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:34.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:34.920 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:34 smithi038 ceph-mon[38113]: pgmap v386: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.51k op/s; 9321/229764 objects misplaced (4.057%); 27 MiB/s, 428 objects/s recovering 2024-03-04T16:11:34.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:34 smithi038 ceph-mon[29779]: pgmap v386: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.51k op/s; 9321/229764 objects misplaced (4.057%); 27 MiB/s, 428 objects/s recovering 2024-03-04T16:11:34.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:34.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:35.100 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:35.133 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:35.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:35.267 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:35.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:35.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:35.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:35.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:35.651 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.683 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:35.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:35.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:35.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:35.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:36.064 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:36.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:36.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:36.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:36.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:36.343 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.379 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:36.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:36.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:36.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:36.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:36.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:36.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:36 smithi038 ceph-mon[29779]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 9321/229764 objects misplaced (4.057%); 24 MiB/s, 377 objects/s recovering 2024-03-04T16:11:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:36 smithi038 ceph-mon[38113]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 9321/229764 objects misplaced (4.057%); 24 MiB/s, 377 objects/s recovering 2024-03-04T16:11:36.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.829 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:36.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:36 smithi176 ceph-mon[33619]: pgmap v387: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 9321/229764 objects misplaced (4.057%); 24 MiB/s, 377 objects/s recovering 2024-03-04T16:11:36.905 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:11:36.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:36.956 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:11:36.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:36.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:37.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:11:37.055 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:11:37.062 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:37.098 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:11:37.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:37.141 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:11:37.188 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:11:37.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:37.235 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:11:37.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:11:37.244 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:11:37.245 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:11:37.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.391 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:37.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:37.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:37.525 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:37.574 INFO:teuthology.orchestra.run.smithi038.stderr:set 4.7 pg_upmap_items mapping to [1->7,5->0,3->6] 2024-03-04T16:11:37.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:37.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.670 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:37.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:37.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:37.803 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1157471340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3861833579' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:37 smithi176 ceph-mon[33619]: from='client.25174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:37.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:37.939 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:37.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1157471340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3861833579' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[29779]: from='client.25174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1157471340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3861833579' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:37.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:37 smithi038 ceph-mon[38113]: from='client.25174 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]: dispatch 2024-03-04T16:11:38.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.076 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:38.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:38.213 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:38.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:38.356 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:38.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.493 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:38.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:38.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[29779]: pgmap v388: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 9321/229764 objects misplaced (4.057%); 19 MiB/s, 310 objects/s recovering 2024-03-04T16:11:38.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[29779]: from='client.25174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]': finished 2024-03-04T16:11:38.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[29779]: osdmap e218: 8 total, 8 up, 6 in 2024-03-04T16:11:38.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[38113]: pgmap v388: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 9321/229764 objects misplaced (4.057%); 19 MiB/s, 310 objects/s recovering 2024-03-04T16:11:38.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[38113]: from='client.25174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]': finished 2024-03-04T16:11:38.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:38 smithi038 ceph-mon[38113]: osdmap e218: 8 total, 8 up, 6 in 2024-03-04T16:11:38.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:38.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:38.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:38 smithi176 ceph-mon[33619]: pgmap v388: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.7 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 9321/229764 objects misplaced (4.057%); 19 MiB/s, 310 objects/s recovering 2024-03-04T16:11:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:38 smithi176 ceph-mon[33619]: from='client.25174 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.7", "id": [1, 7, 5, 0, 3, 6]}]': finished 2024-03-04T16:11:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:38 smithi176 ceph-mon[33619]: osdmap e218: 8 total, 8 up, 6 in 2024-03-04T16:11:38.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:38.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:38.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:38.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:39.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:39.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.176 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:39.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:39.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.313 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:39.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:39.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:39.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:39.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.592 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:39.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:39.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:39.725 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:39.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:39.784 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:39 smithi038 ceph-mon[29779]: osdmap e219: 8 total, 8 up, 6 in 2024-03-04T16:11:39.785 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:39 smithi038 ceph-mon[38113]: osdmap e219: 8 total, 8 up, 6 in 2024-03-04T16:11:39.827 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:39.862 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:39 smithi176 ceph-mon[33619]: osdmap e219: 8 total, 8 up, 6 in 2024-03-04T16:11:39.867 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:39.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:39.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.014 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:40.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:40.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:40.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:40.246 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:11:40.246 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:84324 7.f 7:f790d103:::benchmark_data_smithi038_77863_object84323:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:05:41.191549+0000", 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 359.05083325300001, 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.16557920400000001, 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.247 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 84324 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191549+0000", 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191549+0000", 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.248 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191551+0000", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0480000000000001e-06 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191573+0000", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2597999999999999e-05 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191583+0000", 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4120000000000007e-06 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.249 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.191590+0000", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3100000000000003e-06 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.193621+0000", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020315530000000002 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.193743+0000", 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012123 2024-03-04T16:11:40.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.194564+0000", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00082094299999999998 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.198444+0000", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038806719999999999 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.202012+0000", 2024-03-04T16:11:40.251 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035675770000000002 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.357020+0000", 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.15500803899999999 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.357077+0000", 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7130999999999999e-05 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.252 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:05:41.357128+0000", 2024-03-04T16:11:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0691e-05 2024-03-04T16:11:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192298 7.6 7:6b51513d:::benchmark_data_smithi038_77863_object74030:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.953965689, 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.062137957, 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192298 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.255 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288417+0000", 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2300000000000001e-06 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288422+0000", 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.135e-06 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288433+0000", 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0285000000000001e-05 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.256 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288441+0000", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999993e-06 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.289307+0000", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086642499999999999 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.289315+0000", 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5310000000000004e-06 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.257 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.340852+0000", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0515369299999999 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.340984+0000", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013177999999999999 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350502+0000", 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095186539999999997 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350554+0000", 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1827999999999999e-05 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192320 7.6 7:6d231687:::benchmark_data_smithi038_77863_object74052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.93097842, 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.035261258, 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.259 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192320 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.261 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.261 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311405+0000", 2024-03-04T16:11:40.261 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.04e-06 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311411+0000", 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0859999999999998e-06 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311423+0000", 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1046000000000001e-05 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.262 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311431+0000", 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7099999999999996e-06 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311487+0000", 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6057999999999998e-05 2024-03-04T16:11:40.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311496+0000", 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4209999999999995e-06 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341512+0000", 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0300166609999999 2024-03-04T16:11:40.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341635+0000", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012253399999999999 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341758+0000", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012314800000000001 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346404+0000", 2024-03-04T16:11:40.265 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046457019999999998 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346411+0000", 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6899999999999992e-06 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346508+0000", 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6351e-05 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.266 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346584+0000", 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6139999999999999e-05 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346665+0000", 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0671000000000001e-05 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192361 7.6 7:620642f3:::benchmark_data_smithi038_77863_object74093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.883127285, 2024-03-04T16:11:40.267 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99774576100000001, 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192361 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359255+0000", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.23e-07 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359258+0000", 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2179999999999999e-06 2024-03-04T16:11:40.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359268+0000", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1220000000000001e-06 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359273+0000", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.519e-06 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359428+0000", 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154499 2024-03-04T16:11:40.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359436+0000", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8859999999999993e-06 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346702+0000", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98726572000000001 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346789+0000", 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6763999999999998e-05 2024-03-04T16:11:40.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346861+0000", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2237999999999995e-05 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350594+0000", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037329300000000002 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350651+0000", 2024-03-04T16:11:40.272 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6870999999999998e-05 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356749+0000", 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006097585 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356860+0000", 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111586 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.273 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357000+0000", 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014009999999999999 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192379 7.6 7:642a4bb6:::benchmark_data_smithi038_77863_object74111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.274 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.84985402800001, 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96479910099999999, 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192379 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:11:40.275 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392530+0000", 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4990000000000001e-06 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.276 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392538+0000", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0099999999999995e-06 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392554+0000", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5384e-05 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392562+0000", 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3969999999999994e-06 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392629+0000", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6717999999999994e-05 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392641+0000", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2265e-05 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350689+0000", 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95804780499999997 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350784+0000", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4969000000000005e-05 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350892+0000", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108572 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356719+0000", 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058263000000000004 2024-03-04T16:11:40.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357015+0000", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000296545 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357061+0000", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5828000000000002e-05 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357223+0000", 2024-03-04T16:11:40.280 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016149400000000001 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357327+0000", 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010431500000000001 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192415 7.6 7:6f2667ec:::benchmark_data_smithi038_77863_object74147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.81444331899999, 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93521214699999999, 2024-03-04T16:11:40.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192415 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.282 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427940+0000", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.083e-06 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427944+0000", 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5190000000000003e-06 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427961+0000", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7326e-05 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427969+0000", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4019999999999996e-06 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.428022+0000", 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3430000000000002e-05 2024-03-04T16:11:40.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.428030+0000", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1249999999999993e-06 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353490+0000", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92545976799999996 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353635+0000", 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000145358 2024-03-04T16:11:40.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.286 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.286 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353861+0000", 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000225882 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357393+0000", 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035317090000000001 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357434+0000", 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1405000000000001e-05 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.362975+0000", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055406810000000004 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363045+0000", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9834999999999998e-05 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363151+0000", 2024-03-04T16:11:40.288 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105624 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192419 7.6 7:61106e67:::benchmark_data_smithi038_77863_object74151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.81102209299999, 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93476213799999996, 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192419 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431361+0000", 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1340000000000001e-06 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431365+0000", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3619999999999999e-06 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431375+0000", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0117e-05 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431381+0000", 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0929999999999999e-06 2024-03-04T16:11:40.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431438+0000", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6604000000000002e-05 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431447+0000", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6519999999999995e-06 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357620+0000", 2024-03-04T16:11:40.294 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92617308499999995 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357764+0000", 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014397800000000001 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.358209+0000", 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000445043 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.295 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365924+0000", 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077151900000000002 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365936+0000", 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.225e-05 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365979+0000", 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2342000000000002e-05 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.296 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366027+0000", 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8834e-05 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366122+0000", 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4454000000000001e-05 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192427 7.6 7:63454d35:::benchmark_data_smithi038_77863_object74159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.297 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.80328661799999, 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93643149199999998, 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192427 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:11:40.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439097+0000", 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5549999999999999e-06 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439101+0000", 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1609999999999996e-06 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439109+0000", 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439115+0000", 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3980000000000004e-06 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439176+0000", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1017999999999998e-05 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439183+0000", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000002e-06 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363184+0000", 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92400118200000003 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363330+0000", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146102 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363445+0000", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115333 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371572+0000", 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081264449999999995 2024-03-04T16:11:40.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371613+0000", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1582000000000003e-05 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375274+0000", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036606170000000001 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375427+0000", 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015290900000000001 2024-03-04T16:11:40.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375527+0000", 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9822999999999996e-05 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192435 7.6 7:64907183:::benchmark_data_smithi038_77863_object74167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.79712942099999, 2024-03-04T16:11:40.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93044704600000006, 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192435 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445254+0000", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.525e-06 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445258+0000", 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4139999999999996e-06 2024-03-04T16:11:40.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445266+0000", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6179999999999997e-06 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445272+0000", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1040000000000003e-06 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445342+0000", 2024-03-04T16:11:40.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9524999999999996e-05 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445350+0000", 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9719999999999992e-06 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366160+0000", 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92081084300000005 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366262+0000", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010210200000000001 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366364+0000", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101168 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371648+0000", 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052841750000000003 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375566+0000", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039183600000000001 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375578+0000", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1749999999999999e-05 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375617+0000", 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9529e-05 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375699+0000", 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1960999999999997e-05 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192455 7.6 7:6a9eb3fa:::benchmark_data_smithi038_77863_object74187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.77482847300001, 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92415198300000001, 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192455 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467555+0000", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6559999999999999e-06 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467560+0000", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5929999999999999e-06 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467568+0000", 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3960000000000006e-06 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467574+0000", 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6559999999999997e-06 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467628+0000", 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4486e-05 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467646+0000", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8278e-05 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.368906+0000", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.901259433 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.369025+0000", 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119533 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.369130+0000", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105062 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371681+0000", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025508369999999998 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375706+0000", 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040243689999999999 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391530+0000", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015824181 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391593+0000", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3601000000000003e-05 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391705+0000", 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111902 2024-03-04T16:11:40.317 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192473 7.6 7:66559066:::benchmark_data_smithi038_77863_object74205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.74660696199999, 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.896142141, 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192473 2024-03-04T16:11:40.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495777+0000", 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3609999999999999e-06 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495785+0000", 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999993e-06 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495804+0000", 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8278e-05 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495813+0000", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1130000000000007e-06 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495882+0000", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8935000000000003e-05 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495892+0000", 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9750000000000002e-06 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371709+0000", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87581735599999999 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371819+0000", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109969 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371926+0000", 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010731 2024-03-04T16:11:40.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375711+0000", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037846120000000001 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383485+0000", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077737589999999999 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391753+0000", 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00826817 2024-03-04T16:11:40.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391819+0000", 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6058999999999999e-05 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391917+0000", 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8084999999999993e-05 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192480 7.6 7:623fa8f4:::benchmark_data_smithi038_77863_object74212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.73540021900001, 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88821274100000003, 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192480 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506983+0000", 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.697e-06 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506989+0000", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507003+0000", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3961999999999999e-05 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507011+0000", 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2239999999999995e-06 2024-03-04T16:11:40.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507066+0000", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4805999999999997e-05 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507074+0000", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4819999999999994e-06 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375735+0000", 2024-03-04T16:11:40.328 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86866097200000003 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375842+0000", 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010695499999999999 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375959+0000", 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011725799999999999 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383524+0000", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075642280000000001 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.394497+0000", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010973696 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395111+0000", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061341800000000001 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395156+0000", 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5158999999999999e-05 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395194+0000", 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8393000000000003e-05 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192483 7.6 7:6e5767a6:::benchmark_data_smithi038_77863_object74215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.73164094200001, 2024-03-04T16:11:40.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88546532899999997, 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192483 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510743+0000", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.81e-06 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510749+0000", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5459999999999997e-06 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510764+0000", 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4452e-05 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510773+0000", 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4059999999999994e-06 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510832+0000", 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8909999999999997e-05 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510843+0000", 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1097999999999999e-05 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380203+0000", 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86935975700000001 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380333+0000", 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012970200000000001 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380440+0000", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107753 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383563+0000", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031223760000000001 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395202+0000", 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011639534999999999 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396095+0000", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089278899999999995 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396151+0000", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6094000000000002e-05 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396206+0000", 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5101e-05 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.339 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192486 7.6 7:62008682:::benchmark_data_smithi038_77863_object74218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.72804179400001, 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88626451299999998, 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192486 2024-03-04T16:11:40.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514341+0000", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.443e-06 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514347+0000", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3319999999999999e-06 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514362+0000", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4824000000000001e-05 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514369+0000", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7849999999999992e-06 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514431+0000", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1822999999999994e-05 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514440+0000", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6389999999999998e-06 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383592+0000", 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86915199099999996 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383701+0000", 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109064 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383806+0000", 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010532299999999999 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391470+0000", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076637140000000003 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395208+0000", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003738416 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400506+0000", 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052977520000000002 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400558+0000", 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1894999999999997e-05 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400604+0000", 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6511999999999998e-05 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.346 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192499 7.6 7:653bbb72:::benchmark_data_smithi038_77863_object74231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.713086554, 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.871864572, 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192499 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529297+0000", 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.846e-06 2024-03-04T16:11:40.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529303+0000", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6459999999999998e-06 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529316+0000", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529323+0000", 2024-03-04T16:11:40.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9170000000000006e-06 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529385+0000", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1257e-05 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529396+0000", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1481000000000001e-05 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386219+0000", 2024-03-04T16:11:40.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.856823102 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386368+0000", 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014870500000000001 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386471+0000", 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102537 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.394461+0000", 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079902649999999999 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395213+0000", 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075248199999999996 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401073+0000", 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005859314 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401122+0000", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9418e-05 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401160+0000", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7778000000000003e-05 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192505 7.6 7:618cc37a:::benchmark_data_smithi038_77863_object74237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:11:40.353 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.69312664, 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.85606340400000003, 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192505 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549257+0000", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1679999999999998e-06 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549265+0000", 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1940000000000001e-06 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549280+0000", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5509e-05 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549288+0000", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6079999999999998e-06 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549353+0000", 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5479000000000001e-05 2024-03-04T16:11:40.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549366+0000", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2652e-05 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.388847+0000", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83948117300000002 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.388967+0000", 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120574 2024-03-04T16:11:40.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.389076+0000", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108552 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395055+0000", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059790970000000001 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395218+0000", 2024-03-04T16:11:40.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016246300000000001 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405206+0000", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099885349999999998 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405266+0000", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0186999999999998e-05 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405319+0000", 2024-03-04T16:11:40.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2213000000000003e-05 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192534 7.6 7:60f993e4:::benchmark_data_smithi038_77863_object74266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 237.481532319, 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.64461839600000004, 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.360 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.361 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:40.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:40.362 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:11:40.362 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192534 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.363 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760852+0000", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4669999999999998e-06 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760859+0000", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3039999999999999e-06 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760873+0000", 2024-03-04T16:11:40.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4093999999999999e-05 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760881+0000", 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7060000000000004e-06 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760933+0000", 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2343999999999999e-05 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760941+0000", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8199999999999997e-06 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391958+0000", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.63101682100000001 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.392074+0000", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115573 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:11:40.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.392184+0000", 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011017499999999999 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395093+0000", 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029088970000000001 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396983+0000", 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018902109999999999 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405359+0000", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083764100000000008 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405427+0000", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7448999999999997e-05 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405468+0000", 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1124999999999997e-05 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.368 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114774 8.6 8:694af726:::benchmark_data_smithi038_107057_object114773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 72.012471738000002, 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1166456839999999, 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114774 2024-03-04T16:11:40.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229912+0000", 2024-03-04T16:11:40.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2620000000000001e-06 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229973+0000", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0553000000000001e-05 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229996+0000", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2646999999999998e-05 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230005+0000", 2024-03-04T16:11:40.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4769999999999992e-06 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230092+0000", 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6667000000000002e-05 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230253+0000", 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016137700000000001 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,7", 2024-03-04T16:11:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230416+0000", 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016295200000000001 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243867+0000", 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013450632000000001 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243882+0000", 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5690999999999999e-05 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232356+0000", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98847337300000004 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232375+0000", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9768000000000001e-05 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.330696+0000", 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.098320972 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331886+0000", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011898729999999999 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346472+0000", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014585653000000001 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346556+0000", 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3787000000000003e-05 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.375 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114798 8.6 8:6e193aa2:::benchmark_data_smithi038_107057_object114797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.790263952999993, 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.89634460400000004, 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114798 2024-03-04T16:11:40.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452121+0000", 2024-03-04T16:11:40.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.022e-06 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452154+0000", 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2851000000000001e-05 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452177+0000", 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2880000000000001e-05 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452185+0000", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8149999999999995e-06 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452277+0000", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1267999999999994e-05 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452292+0000", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5811000000000002e-05 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332203+0000", 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87991063700000005 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332343+0000", 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013942199999999999 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332826+0000", 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048315199999999998 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346595+0000", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013768927 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346632+0000", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7630999999999999e-05 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346680+0000", 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7589999999999997e-05 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348078+0000", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013984640000000001 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348149+0000", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0673000000000006e-05 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348462+0000", 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031346100000000001 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.382 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114804 8.6 8:642351e4:::benchmark_data_smithi038_107057_object114803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 70.768638968000005, 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87523897799999995, 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114804 2024-03-04T16:11:40.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473746+0000", 2024-03-04T16:11:40.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8600000000000001e-06 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473780+0000", 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4461000000000001e-05 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473798+0000", 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8297000000000001e-05 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:40.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473809+0000", 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473890+0000", 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0513999999999995e-05 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473913+0000", 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3564e-05 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343022+0000", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86910901200000001 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343537+0000", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051479200000000003 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343752+0000", 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000215195 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346685+0000", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002932933 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.347988+0000", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013024550000000001 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348035+0000", 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6859e-05 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348780+0000", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074556000000000004 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348845+0000", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4453000000000001e-05 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348982+0000", 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013713899999999999 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.389 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:40.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:40.390 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:11:40.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.502 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:40.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:40.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:40.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:40.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:40.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:40.779 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: osdmap e220: 8 total, 8 up, 6 in 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/804622627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:40 smithi176 ceph-mon[33619]: from='client.25183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:40.881 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:40.915 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:40.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: osdmap e220: 8 total, 8 up, 6 in 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/804622627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[29779]: from='client.25183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: pgmap v391: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: osdmap e220: 8 total, 8 up, 6 in 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/804622627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:40 smithi038 ceph-mon[38113]: from='client.25183 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:11:41.016 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:41.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:41.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:11:41.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.189 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:41.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:41.236 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:11:41.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:41.322 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:41.424 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:41.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:41.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.586 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:41.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:41.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.729 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:41.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:41.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:41 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:41 smithi176 ceph-mon[33619]: from='client.25183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:41 smithi176 ceph-mon[33619]: osdmap e221: 8 total, 8 up, 6 in 2024-03-04T16:11:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:41.874 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:41.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:41.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[29779]: from='client.25183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[29779]: osdmap e221: 8 total, 8 up, 6 in 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[38113]: from='client.25183 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:11:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:41 smithi038 ceph-mon[38113]: osdmap e221: 8 total, 8 up, 6 in 2024-03-04T16:11:42.008 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:42.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:42.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:42.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:11:42.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:42.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:42.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:42.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:42.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:42.602 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:42.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.746 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:42.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:42.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:42.883 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:42.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:42.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:43.016 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:43.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:43.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:43.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:43.175 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:43.219 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:43.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:43.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:43.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:43.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:43 smithi038 ceph-mon[29779]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:43.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:43 smithi038 ceph-mon[38113]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:43.312 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:43.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:43.436 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:43.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.572 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:43.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:43 smithi176 ceph-mon[33619]: pgmap v394: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 7426/211164 objects misplaced (3.517%); 19 MiB/s, 303 objects/s recovering 2024-03-04T16:11:43.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.715 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:43.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:43.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:43.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:43.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:43.995 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:43.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:44.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.131 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:44.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:44.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:44.269 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:44.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.406 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:44.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:44.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:44 smithi038 ceph-mon[29779]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 1.75k op/s; 5536/191733 objects misplaced (2.887%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:11:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:44.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:44 smithi038 ceph-mon[38113]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 1.75k op/s; 5536/191733 objects misplaced (2.887%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:11:44.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:44.554 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:44.572 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:11:44.572 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:11:44.572 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-04T16:11:44.573 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-04T16:11:44.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:11:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:44.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:44 smithi176 ceph-mon[33619]: pgmap v395: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 866 B/s rd, 0 B/s wr, 1.75k op/s; 5536/191733 objects misplaced (2.887%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:11:44.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:44.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:44.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:44.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:44.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:44.965 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:44.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:45.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:45.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:45.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:11:45.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:45.235 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:45.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:45.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:45.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.364 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:45.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:45.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:45.509 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:45.611 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:45.635 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:45.737 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.773 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:45.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:45.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:45.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:45.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:46.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:46.052 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:46.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:46.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:46.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.391 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:46.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:46.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:46 smithi038 ceph-mon[38113]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 939 op/s; 5536/191733 objects misplaced (2.887%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:11:46.417 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:46.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:46 smithi038 ceph-mon[29779]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 939 op/s; 5536/191733 objects misplaced (2.887%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:11:46.418 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:46.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:46.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:46 smithi176 ceph-mon[33619]: pgmap v396: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 939 op/s; 5536/191733 objects misplaced (2.887%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:11:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:11:46.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:46.689 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:46.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:184362 8.3 8:ccceb9bb:::benchmark_data_smithi038_107057_object62727:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:46.788182+0000", 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0083453160000000002, 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083707680000000007, 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:11:46.799 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 184362 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788182+0000", 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:46.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788182+0000", 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788183+0000", 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.849e-06 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788189+0000", 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8869999999999997e-06 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788200+0000", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0545e-05 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788207+0000", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5789999999999998e-06 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788404+0000", 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019651900000000001 2024-03-04T16:11:46.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788549+0000", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000145136 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788695+0000", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146055 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794607+0000", 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059121269999999997 2024-03-04T16:11:46.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794660+0000", 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2441000000000002e-05 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794722+0000", 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2228000000000001e-05 2024-03-04T16:11:46.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:184367 8.3 8:c39fbbfc:::benchmark_data_smithi038_107057_object62732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:46.788244+0000", 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0082825099999999999, 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083689939999999994, 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:11:46.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 184367 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788244+0000", 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:46.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788244+0000", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788245+0000", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4000000000000002e-07 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788247+0000", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6010000000000002e-06 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:46.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788253+0000", 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1000000000000003e-06 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.788258+0000", 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9409999999999998e-06 2024-03-04T16:11:46.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:46.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.791006+0000", 2024-03-04T16:11:46.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002748646 2024-03-04T16:11:46.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:46.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.791151+0000", 2024-03-04T16:11:46.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014507100000000001 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.791313+0000", 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016149599999999999 2024-03-04T16:11:46.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794729+0000", 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034161759999999999 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:46.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15171.0:184360 8.0 e221/167 8:0834a357:::benchmark_data_smithi038_107057_object62725:head v 221'11290, mlcod=221'11287)", 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:11:46.794316+0000", 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0022109019999999998, 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002345195, 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:11:46.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794316+0000", 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794316+0000", 2024-03-04T16:11:46.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794317+0000", 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5510000000000001e-06 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794325+0000", 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9249999999999995e-06 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:11:46.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794336+0000", 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1045e-05 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794344+0000", 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6159999999999996e-06 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794401+0000", 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6530000000000003e-05 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:11:46.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:11:46.794422+0000", 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1716999999999999e-05 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 3 2024-03-04T16:11:46.817 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:11:46.822 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:46.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:46.961 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:46.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:47.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:47.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:47.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:47.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:47.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:47.379 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:47.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:47.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:47.527 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:11:47.566 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:11:47.595 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:11:47.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:47.626 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:11:47.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:47.663 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:11:47.687 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:11:47.712 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:11:47.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:47.740 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:11:47.742 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:11:47.762 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:47.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:47.888 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:47.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:47.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:48.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:48.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.142 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:48.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:48.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:48.272 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:48.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:48.410 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:48.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:48.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: osdmap e222: 8 total, 8 up, 6 in 2024-03-04T16:11:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 649 B/s rd, 0 B/s wr, 822 op/s; 5536/191733 objects misplaced (2.887%); 9.9 MiB/s, 158 objects/s recovering 2024-03-04T16:11:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2587618754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:48.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:48 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:48.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:48.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: osdmap e222: 8 total, 8 up, 6 in 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 649 B/s rd, 0 B/s wr, 822 op/s; 5536/191733 objects misplaced (2.887%); 9.9 MiB/s, 158 objects/s recovering 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2587618754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: osdmap e222: 8 total, 8 up, 6 in 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: pgmap v398: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 649 B/s rd, 0 B/s wr, 822 op/s; 5536/191733 objects misplaced (2.887%); 9.9 MiB/s, 158 objects/s recovering 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2587618754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:11:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:48 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:48.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:48.822 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:48.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:11:48.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:48.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:48.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:49.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:49.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.085 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:49.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.220 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:49.321 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.357 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:49.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:49.487 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:49.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:49 smithi038 ceph-mon[29779]: osdmap e223: 8 total, 8 up, 6 in 2024-03-04T16:11:49.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:49 smithi038 ceph-mon[38113]: osdmap e223: 8 total, 8 up, 6 in 2024-03-04T16:11:49.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:49 smithi176 ceph-mon[33619]: osdmap e223: 8 total, 8 up, 6 in 2024-03-04T16:11:49.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:49.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:49.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:49.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:49.869 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:49.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:49.997 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:49.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:50.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:50.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:50.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:50.253 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:50.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.390 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:50.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:50.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:50.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:50 smithi176 ceph-mon[33619]: osdmap e224: 8 total, 8 up, 6 in 2024-03-04T16:11:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:50 smithi176 ceph-mon[33619]: pgmap v401: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 928 op/s; 3350/175032 objects misplaced (1.914%); 19 MiB/s, 296 objects/s recovering 2024-03-04T16:11:50.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:50.672 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:50 smithi038 ceph-mon[29779]: osdmap e224: 8 total, 8 up, 6 in 2024-03-04T16:11:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:50 smithi038 ceph-mon[29779]: pgmap v401: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 928 op/s; 3350/175032 objects misplaced (1.914%); 19 MiB/s, 296 objects/s recovering 2024-03-04T16:11:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:50 smithi038 ceph-mon[38113]: osdmap e224: 8 total, 8 up, 6 in 2024-03-04T16:11:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:50 smithi038 ceph-mon[38113]: pgmap v401: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 928 op/s; 3350/175032 objects misplaced (1.914%); 19 MiB/s, 296 objects/s recovering 2024-03-04T16:11:50.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:50.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:50.917 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:50.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:50.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:51.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:51.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:51.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:51.222 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:51.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:51.364 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:51.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:51.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:51.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:51.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:51 smithi176 ceph-mon[33619]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-04T16:11:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:51 smithi176 ceph-mon[33619]: osdmap e225: 8 total, 8 up, 6 in 2024-03-04T16:11:51.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:51.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:51.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:51 smithi038 ceph-mon[29779]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-04T16:11:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:51 smithi038 ceph-mon[29779]: osdmap e225: 8 total, 8 up, 6 in 2024-03-04T16:11:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:51 smithi038 ceph-mon[38113]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-04T16:11:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:51 smithi038 ceph-mon[38113]: osdmap e225: 8 total, 8 up, 6 in 2024-03-04T16:11:51.767 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:51.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:51.907 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:52.009 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:52.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:52.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:11:52.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.193 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:52.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:52.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:52.328 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:52.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:52.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:52.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:52.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.612 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:52.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:52.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:52.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:52.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:52.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:52.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:52.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:53.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:53.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:11:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:11:53.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.163 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:53.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:53.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:53.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:53.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[29779]: pgmap v403: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 3350/175032 objects misplaced (1.914%); 26 MiB/s, 423 objects/s recovering 2024-03-04T16:11:53.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:53.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:53.367 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[38113]: pgmap v403: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 3350/175032 objects misplaced (1.914%); 26 MiB/s, 423 objects/s recovering 2024-03-04T16:11:53.367 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:53.367 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:53.368 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:11:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:11:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:11:53.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:53.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:53.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:53.575 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:53 smithi176 ceph-mon[33619]: pgmap v403: 148 pgs: 1 peering, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 3350/175032 objects misplaced (1.914%); 26 MiB/s, 423 objects/s recovering 2024-03-04T16:11:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:11:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:11:53.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.711 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:53.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:53.812 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:53.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.846 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:53.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:53.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:53.968 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:11:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:53.984 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:53.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:54.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:54.129 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:54.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:54.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:54.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:54.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:54.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.413 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:54.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:54.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:54.543 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:11:54.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:54.684 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:54.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:54.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:54.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:54.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:54.947 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:54.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:54.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:55.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:55.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:55.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.209 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:55.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:55.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:55.349 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:55.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:55.451 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:55.481 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:55.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:55 smithi038 ceph-mon[29779]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.77k op/s; 1974/159780 objects misplaced (1.235%); 30 MiB/s, 485 objects/s recovering 2024-03-04T16:11:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:55 smithi038 ceph-mon[38113]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.77k op/s; 1974/159780 objects misplaced (1.235%); 30 MiB/s, 485 objects/s recovering 2024-03-04T16:11:55.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:55 smithi176 ceph-mon[33619]: pgmap v404: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.77k op/s; 1974/159780 objects misplaced (1.235%); 30 MiB/s, 485 objects/s recovering 2024-03-04T16:11:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:55.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:55.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:55.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:55.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:55.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:55.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:55.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:56.024 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:56.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:56.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:56.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:56.294 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:56.396 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:56.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:56.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:56 smithi038 ceph-mon[29779]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 709 B/s rd, 0 B/s wr, 837 op/s; 1974/159780 objects misplaced (1.235%); 9.8 MiB/s, 156 objects/s recovering 2024-03-04T16:11:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:56 smithi038 ceph-mon[38113]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 709 B/s rd, 0 B/s wr, 837 op/s; 1974/159780 objects misplaced (1.235%); 9.8 MiB/s, 156 objects/s recovering 2024-03-04T16:11:56.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.563 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:56.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:56 smithi176 ceph-mon[33619]: pgmap v405: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 709 B/s rd, 0 B/s wr, 837 op/s; 1974/159780 objects misplaced (1.235%); 9.8 MiB/s, 156 objects/s recovering 2024-03-04T16:11:56.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:56.698 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:56.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:56.828 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:56.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:56.961 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:56.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:57.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.107 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:57.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:57.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:57.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:57.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:57.276 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:11:57.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:57.379 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:57.481 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:57.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:57.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:57 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:11:57.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:11:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:11:57.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:57.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:57.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:57.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:57.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:57.936 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:57.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:58.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:58.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:58.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:11:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:11:58.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.345 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:58.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:58.446 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:58.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:58.480 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:58.480 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: osdmap e226: 8 total, 8 up, 6 in 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: pgmap v407: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 1974/159780 objects misplaced (1.235%); 8.8 MiB/s, 141 objects/s recovering 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: osdmap e226: 8 total, 8 up, 6 in 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: pgmap v407: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 1974/159780 objects misplaced (1.235%); 8.8 MiB/s, 141 objects/s recovering 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:11:58.481 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:58 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:11:58.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:11:58.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3276946834' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: osdmap e226: 8 total, 8 up, 6 in 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: pgmap v407: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 3.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 635 op/s; 1974/159780 objects misplaced (1.235%); 8.8 MiB/s, 141 objects/s recovering 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:11:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:58 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:11:58.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:58.753 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:58.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:58.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:58.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:58.961 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:11:58.963 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:11:58.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:59.027 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:11:59.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:11:59.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:11:59.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:59.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:59.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:11:59.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:11:59.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:11:59.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:11:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:11:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:59 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:11:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:59 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:11:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:59 smithi176 ceph-mon[33619]: osdmap e227: 8 total, 8 up, 6 in 2024-03-04T16:11:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:59 smithi176 ceph-mon[33619]: 8.c scrub starts 2024-03-04T16:11:59.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:11:59 smithi176 ceph-mon[33619]: from='client.15345 v1:172.21.15.38:0/2336301329' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:59.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:11:59.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:11:59.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[29779]: osdmap e227: 8 total, 8 up, 6 in 2024-03-04T16:11:59.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[29779]: 8.c scrub starts 2024-03-04T16:11:59.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[29779]: from='client.15345 v1:172.21.15.38:0/2336301329' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:59.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:11:59.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:11:59.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[38113]: osdmap e227: 8 total, 8 up, 6 in 2024-03-04T16:11:59.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[38113]: 8.c scrub starts 2024-03-04T16:11:59.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:11:59 smithi038 ceph-mon[38113]: from='client.15345 v1:172.21.15.38:0/2336301329' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:11:59.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:11:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.724 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:59.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:11:59.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:11:59.858 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:11:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:11:59.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:11:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:11:59.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:00.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:00.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:00.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:00.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:00.274 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:00.277 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:00.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:00.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.427 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:00.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:00.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:00.565 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:00.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1372/140007 objects misplaced (0.980%); 12 MiB/s, 191 objects/s recovering 2024-03-04T16:12:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:00 smithi176 ceph-mon[33619]: 8.c scrub ok 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1372/140007 objects misplaced (0.980%); 12 MiB/s, 191 objects/s recovering 2024-03-04T16:12:00.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[29779]: 8.c scrub ok 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: pgmap v409: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 1372/140007 objects misplaced (0.980%); 12 MiB/s, 191 objects/s recovering 2024-03-04T16:12:00.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:00 smithi038 ceph-mon[38113]: 8.c scrub ok 2024-03-04T16:12:00.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.707 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:00.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:00.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.841 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:00.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:00.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:00.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:00.973 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:00.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:00.989 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-03-04T16:11:58.319666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3125090678,"num_objects":47875,"num_object_clones":0,"num_object_copies":143625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1372,"num_objects_unfound":0,"num_objects_dirty":47875,"num_whiteouts":0,"num_read":1467,"num_read_kb":1313,"num_write":196204,"num_write_kb":7786910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66351,"num_bytes_recovered":4334365994,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65970,"ondisk_log_size":65970,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":11422092,"kb_used_data":9238284,"kb_used_omap":9,"kb_used_meta":2183350,"kb_avail":551048820,"statfs":{"total":575970213888,"available":564273991680,"internally_reserved":0,"allocated":9460002816,"data_stored":9457639815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":2235751079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-686096384,"num_objects":-10469,"num_object_clones":0,"num_object_copies":-31407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1978,"num_objects_unfound":0,"num_objects_dirty":-10469,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10469,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1536,"num_bytes_recovered":100597760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.030851"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":350,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267888+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:11:57.267888+0000","last_peered":"2024-03-04T16:11:57.267888+0000","last_clean":"2024-03-04T16:11:57.267888+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:11:57.267888+0000","last_undegraded":"2024-03-04T16:11:57.267888+0000","last_fullsized":"2024-03-04T16:11:57.267888+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":346,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.281053+0000","last_change":"2024-03-04T16:08:23.797046+0000","last_active":"2024-03-04T16:11:57.281053+0000","last_peered":"2024-03-04T16:11:57.281053+0000","last_clean":"2024-03-04T16:11:57.281053+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:11:57.281053+0000","last_undegraded":"2024-03-04T16:11:57.281053+0000","last_fullsized":"2024-03-04T16:11:57.281053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.794378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274482+0000","last_change":"2024-03-04T16:10:29.339688+0000","last_active":"2024-03-04T16:11:57.274482+0000","last_peered":"2024-03-04T16:11:57.274482+0000","last_clean":"2024-03-04T16:11:57.274482+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:11:57.274482+0000","last_undegraded":"2024-03-04T16:11:57.274482+0000","last_fullsized":"2024-03-04T16:11:57.274482+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.062292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.062292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"75'9","reported_seq":390,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267195+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:11:57.267195+0000","last_peered":"2024-03-04T16:11:57.267195+0000","last_clean":"2024-03-04T16:11:57.267195+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:11:57.267195+0000","last_undegraded":"2024-03-04T16:11:57.267195+0000","last_fullsized":"2024-03-04T16:11:57.267195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":346,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333815+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:11:50.333815+0000","last_peered":"2024-03-04T16:11:50.333815+0000","last_clean":"2024-03-04T16:11:50.333815+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:11:50.333815+0000","last_undegraded":"2024-03-04T16:11:50.333815+0000","last_fullsized":"2024-03-04T16:11:50.333815+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":352,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267202+0000","last_change":"2024-03-04T16:10:29.325452+0000","last_active":"2024-03-04T16:11:57.267202+0000","last_peered":"2024-03-04T16:11:57.267202+0000","last_clean":"2024-03-04T16:11:57.267202+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:11:57.267202+0000","last_undegraded":"2024-03-04T16:11:57.267202+0000","last_fullsized":"2024-03-04T16:11:57.267202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:00.760231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:00.760231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":357,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.334322+0000","last_change":"2024-03-04T16:10:29.251726+0000","last_active":"2024-03-04T16:11:50.334322+0000","last_peered":"2024-03-04T16:11:50.334322+0000","last_clean":"2024-03-04T16:11:50.334322+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:11:50.334322+0000","last_undegraded":"2024-03-04T16:11:50.334322+0000","last_fullsized":"2024-03-04T16:11:50.334322+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"75'15","reported_seq":439,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333273+0000","last_change":"2024-03-04T16:10:17.503786+0000","last_active":"2024-03-04T16:11:50.333273+0000","last_peered":"2024-03-04T16:11:50.333273+0000","last_clean":"2024-03-04T16:11:50.333273+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:11:50.333273+0000","last_undegraded":"2024-03-04T16:11:50.333273+0000","last_fullsized":"2024-03-04T16:11:50.333273+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:21.634651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.634651+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":198,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274854+0000","last_change":"2024-03-04T16:09:12.270171+0000","last_active":"2024-03-04T16:11:57.274854+0000","last_peered":"2024-03-04T16:11:57.274854+0000","last_clean":"2024-03-04T16:11:57.274854+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:11:57.274854+0000","last_undegraded":"2024-03-04T16:11:57.274854+0000","last_fullsized":"2024-03-04T16:11:57.274854+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.270043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.270043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361142+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":345,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273594+0000","last_change":"2024-03-04T16:08:15.131493+0000","last_active":"2024-03-04T16:11:57.273594+0000","last_peered":"2024-03-04T16:11:57.273594+0000","last_clean":"2024-03-04T16:11:57.273594+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:11:57.273594+0000","last_undegraded":"2024-03-04T16:11:57.273594+0000","last_fullsized":"2024-03-04T16:11:57.273594+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:15.131385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:08:15.131385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488415+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":367,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273912+0000","last_change":"2024-03-04T16:10:29.368250+0000","last_active":"2024-03-04T16:11:57.273912+0000","last_peered":"2024-03-04T16:11:57.273912+0000","last_clean":"2024-03-04T16:11:57.273912+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:11:57.273912+0000","last_undegraded":"2024-03-04T16:11:57.273912+0000","last_fullsized":"2024-03-04T16:11:57.273912+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:18.149919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:18.149919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"90'13","reported_seq":402,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268172+0000","last_change":"2024-03-04T16:08:22.740958+0000","last_active":"2024-03-04T16:11:57.268172+0000","last_peered":"2024-03-04T16:11:57.268172+0000","last_clean":"2024-03-04T16:11:57.268172+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:11:57.268172+0000","last_undegraded":"2024-03-04T16:11:57.268172+0000","last_fullsized":"2024-03-04T16:11:57.268172+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.740850+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":357,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273131+0000","last_change":"2024-03-04T16:10:02.242059+0000","last_active":"2024-03-04T16:11:57.273131+0000","last_peered":"2024-03-04T16:11:57.273131+0000","last_clean":"2024-03-04T16:11:57.273131+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:11:57.273131+0000","last_undegraded":"2024-03-04T16:11:57.273131+0000","last_fullsized":"2024-03-04T16:11:57.273131+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.241922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.241922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488864+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269554+0000","last_change":"2024-03-04T16:10:08.741643+0000","last_active":"2024-03-04T16:11:57.269554+0000","last_peered":"2024-03-04T16:11:57.269554+0000","last_clean":"2024-03-04T16:11:57.269554+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:11:57.269554+0000","last_undegraded":"2024-03-04T16:11:57.269554+0000","last_fullsized":"2024-03-04T16:11:57.269554+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:08.741499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:08.741499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275438+0000","last_change":"2024-03-04T16:10:00.097245+0000","last_active":"2024-03-04T16:11:57.275438+0000","last_peered":"2024-03-04T16:11:57.275438+0000","last_clean":"2024-03-04T16:11:57.275438+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:11:57.275438+0000","last_undegraded":"2024-03-04T16:11:57.275438+0000","last_fullsized":"2024-03-04T16:11:57.275438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.097027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.097027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"170'7","reported_seq":386,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268574+0000","last_change":"2024-03-04T16:10:17.490328+0000","last_active":"2024-03-04T16:11:57.268574+0000","last_peered":"2024-03-04T16:11:57.268574+0000","last_clean":"2024-03-04T16:11:57.268574+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:11:57.268574+0000","last_undegraded":"2024-03-04T16:11:57.268574+0000","last_fullsized":"2024-03-04T16:11:57.268574+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:09:04.791092+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.791092+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":349,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268715+0000","last_change":"2024-03-04T16:10:29.330112+0000","last_active":"2024-03-04T16:11:57.268715+0000","last_peered":"2024-03-04T16:11:57.268715+0000","last_clean":"2024-03-04T16:11:57.268715+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:11:57.268715+0000","last_undegraded":"2024-03-04T16:11:57.268715+0000","last_fullsized":"2024-03-04T16:11:57.268715+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:03.790665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:03.790665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.283437+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:11:57.283437+0000","last_peered":"2024-03-04T16:11:57.283437+0000","last_clean":"2024-03-04T16:11:57.283437+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:11:57.283437+0000","last_undegraded":"2024-03-04T16:11:57.283437+0000","last_fullsized":"2024-03-04T16:11:57.283437+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":190,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275306+0000","last_change":"2024-03-04T16:09:11.207728+0000","last_active":"2024-03-04T16:11:57.275306+0000","last_peered":"2024-03-04T16:11:57.275306+0000","last_clean":"2024-03-04T16:11:57.275306+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:11:57.275306+0000","last_undegraded":"2024-03-04T16:11:57.275306+0000","last_fullsized":"2024-03-04T16:11:57.275306+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.207608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.207608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"75'15","reported_seq":142,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278541+0000","last_change":"2024-03-04T16:10:17.515216+0000","last_active":"2024-03-04T16:11:57.278541+0000","last_peered":"2024-03-04T16:11:57.278541+0000","last_clean":"2024-03-04T16:11:57.278541+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:11:57.278541+0000","last_undegraded":"2024-03-04T16:11:57.278541+0000","last_fullsized":"2024-03-04T16:11:57.278541+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:10:01.944401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.944401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":347,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.335406+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:11:50.335406+0000","last_peered":"2024-03-04T16:11:50.335406+0000","last_clean":"2024-03-04T16:11:50.335406+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:11:50.335406+0000","last_undegraded":"2024-03-04T16:11:50.335406+0000","last_fullsized":"2024-03-04T16:11:50.335406+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268313+0000","last_change":"2024-03-04T16:10:17.459301+0000","last_active":"2024-03-04T16:11:57.268313+0000","last_peered":"2024-03-04T16:11:57.268313+0000","last_clean":"2024-03-04T16:11:57.268313+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:11:57.268313+0000","last_undegraded":"2024-03-04T16:11:57.268313+0000","last_fullsized":"2024-03-04T16:11:57.268313+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.804222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.804222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":368,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268205+0000","last_change":"2024-03-04T16:10:17.481021+0000","last_active":"2024-03-04T16:11:57.268205+0000","last_peered":"2024-03-04T16:11:57.268205+0000","last_clean":"2024-03-04T16:11:57.268205+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:11:57.268205+0000","last_undegraded":"2024-03-04T16:11:57.268205+0000","last_fullsized":"2024-03-04T16:11:57.268205+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"75'12","reported_seq":371,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333943+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:11:50.333943+0000","last_peered":"2024-03-04T16:11:50.333943+0000","last_clean":"2024-03-04T16:11:50.333943+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:11:50.333943+0000","last_undegraded":"2024-03-04T16:11:50.333943+0000","last_fullsized":"2024-03-04T16:11:50.333943+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269939+0000","last_change":"2024-03-04T16:11:09.910387+0000","last_active":"2024-03-04T16:11:57.269939+0000","last_peered":"2024-03-04T16:11:57.269939+0000","last_clean":"2024-03-04T16:11:57.269939+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:11:57.269939+0000","last_undegraded":"2024-03-04T16:11:57.269939+0000","last_fullsized":"2024-03-04T16:11:57.269939+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:11:09.910270+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:11:09.910270+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:16.624732+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":290,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269015+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:11:57.269015+0000","last_peered":"2024-03-04T16:11:57.269015+0000","last_clean":"2024-03-04T16:11:57.269015+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:11:57.269015+0000","last_undegraded":"2024-03-04T16:11:57.269015+0000","last_fullsized":"2024-03-04T16:11:57.269015+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.281249+0000","last_change":"2024-03-04T16:08:11.748109+0000","last_active":"2024-03-04T16:11:57.281249+0000","last_peered":"2024-03-04T16:11:57.281249+0000","last_clean":"2024-03-04T16:11:57.281249+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:11:57.281249+0000","last_undegraded":"2024-03-04T16:11:57.281249+0000","last_fullsized":"2024-03-04T16:11:57.281249+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:08:11.747995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:11.747995+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"75'10","reported_seq":211,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274831+0000","last_change":"2024-03-04T16:09:58.487155+0000","last_active":"2024-03-04T16:11:57.274831+0000","last_peered":"2024-03-04T16:11:57.274831+0000","last_clean":"2024-03-04T16:11:57.274831+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:11:57.274831+0000","last_undegraded":"2024-03-04T16:11:57.274831+0000","last_fullsized":"2024-03-04T16:11:57.274831+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:58.487024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.487024+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":342,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.283579+0000","last_change":"2024-03-04T16:08:12.769052+0000","last_active":"2024-03-04T16:11:57.283579+0000","last_peered":"2024-03-04T16:11:57.283579+0000","last_clean":"2024-03-04T16:11:57.283579+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:11:57.283579+0000","last_undegraded":"2024-03-04T16:11:57.283579+0000","last_fullsized":"2024-03-04T16:11:57.283579+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:12.768937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.768937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274998+0000","last_change":"2024-03-04T16:10:17.479793+0000","last_active":"2024-03-04T16:11:57.274998+0000","last_peered":"2024-03-04T16:11:57.274998+0000","last_clean":"2024-03-04T16:11:57.274998+0000","last_became_active":"2024-03-04T16:10:17.478936+0000","last_became_peered":"2024-03-04T16:10:17.478936+0000","last_unstale":"2024-03-04T16:11:57.274998+0000","last_undegraded":"2024-03-04T16:11:57.274998+0000","last_fullsized":"2024-03-04T16:11:57.274998+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.127550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.127550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":364,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.332842+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:11:50.332842+0000","last_peered":"2024-03-04T16:11:50.332842+0000","last_clean":"2024-03-04T16:11:50.332842+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:11:50.332842+0000","last_undegraded":"2024-03-04T16:11:50.332842+0000","last_fullsized":"2024-03-04T16:11:50.332842+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"75'11","reported_seq":124,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273443+0000","last_change":"2024-03-04T16:10:29.384325+0000","last_active":"2024-03-04T16:11:57.273443+0000","last_peered":"2024-03-04T16:11:57.273443+0000","last_clean":"2024-03-04T16:11:57.273443+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:11:57.273443+0000","last_undegraded":"2024-03-04T16:11:57.273443+0000","last_fullsized":"2024-03-04T16:11:57.273443+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:08:58.094130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.094130+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269464+0000","last_change":"2024-03-04T16:10:17.457894+0000","last_active":"2024-03-04T16:11:57.269464+0000","last_peered":"2024-03-04T16:11:57.269464+0000","last_clean":"2024-03-04T16:11:57.269464+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:11:57.269464+0000","last_undegraded":"2024-03-04T16:11:57.269464+0000","last_fullsized":"2024-03-04T16:11:57.269464+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:09:19.779373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267114+0000","last_change":"2024-03-04T16:10:29.238069+0000","last_active":"2024-03-04T16:11:57.267114+0000","last_peered":"2024-03-04T16:11:57.267114+0000","last_clean":"2024-03-04T16:11:57.267114+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:11:57.267114+0000","last_undegraded":"2024-03-04T16:11:57.267114+0000","last_fullsized":"2024-03-04T16:11:57.267114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":190,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274436+0000","last_change":"2024-03-04T16:08:59.144763+0000","last_active":"2024-03-04T16:11:57.274436+0000","last_peered":"2024-03-04T16:11:57.274436+0000","last_clean":"2024-03-04T16:11:57.274436+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:11:57.274436+0000","last_undegraded":"2024-03-04T16:11:57.274436+0000","last_fullsized":"2024-03-04T16:11:57.274436+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:59.144686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.144686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"75'4","reported_seq":366,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.332934+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:11:50.332934+0000","last_peered":"2024-03-04T16:11:50.332934+0000","last_clean":"2024-03-04T16:11:50.332934+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:11:50.332934+0000","last_undegraded":"2024-03-04T16:11:50.332934+0000","last_fullsized":"2024-03-04T16:11:50.332934+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":350,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333918+0000","last_change":"2024-03-04T16:10:29.250985+0000","last_active":"2024-03-04T16:11:50.333918+0000","last_peered":"2024-03-04T16:11:50.333918+0000","last_clean":"2024-03-04T16:11:50.333918+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:11:50.333918+0000","last_undegraded":"2024-03-04T16:11:50.333918+0000","last_fullsized":"2024-03-04T16:11:50.333918+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:06.354675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.356853+0000","last_clean_scrub_stamp":"2024-03-04T16:08:06.354675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269583+0000","last_change":"2024-03-04T16:10:48.950468+0000","last_active":"2024-03-04T16:11:57.269583+0000","last_peered":"2024-03-04T16:11:57.269583+0000","last_clean":"2024-03-04T16:11:57.269583+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:11:57.269583+0000","last_undegraded":"2024-03-04T16:11:57.269583+0000","last_fullsized":"2024-03-04T16:11:57.269583+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:48.950175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:48.950175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:57.893914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273785+0000","last_change":"2024-03-04T16:08:22.164119+0000","last_active":"2024-03-04T16:11:57.273785+0000","last_peered":"2024-03-04T16:11:57.273785+0000","last_clean":"2024-03-04T16:11:57.273785+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:11:57.273785+0000","last_undegraded":"2024-03-04T16:11:57.273785+0000","last_fullsized":"2024-03-04T16:11:57.273785+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:22.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:22.164028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"75'11","reported_seq":385,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333177+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:11:50.333177+0000","last_peered":"2024-03-04T16:11:50.333177+0000","last_clean":"2024-03-04T16:11:50.333177+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:11:50.333177+0000","last_undegraded":"2024-03-04T16:11:50.333177+0000","last_fullsized":"2024-03-04T16:11:50.333177+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":190,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274960+0000","last_change":"2024-03-04T16:09:10.176051+0000","last_active":"2024-03-04T16:11:57.274960+0000","last_peered":"2024-03-04T16:11:57.274960+0000","last_clean":"2024-03-04T16:11:57.274960+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:11:57.274960+0000","last_undegraded":"2024-03-04T16:11:57.274960+0000","last_fullsized":"2024-03-04T16:11:57.274960+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.175912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361111+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":348,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333691+0000","last_change":"2024-03-04T16:08:31.755372+0000","last_active":"2024-03-04T16:11:50.333691+0000","last_peered":"2024-03-04T16:11:50.333691+0000","last_clean":"2024-03-04T16:11:50.333691+0000","last_became_active":"2024-03-04T16:07:36.223300+0000","last_became_peered":"2024-03-04T16:07:36.223300+0000","last_unstale":"2024-03-04T16:11:50.333691+0000","last_undegraded":"2024-03-04T16:11:50.333691+0000","last_fullsized":"2024-03-04T16:11:50.333691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270303+0000","last_change":"2024-03-04T16:10:40.920094+0000","last_active":"2024-03-04T16:11:57.270303+0000","last_peered":"2024-03-04T16:11:57.270303+0000","last_clean":"2024-03-04T16:11:57.270303+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:11:57.270303+0000","last_undegraded":"2024-03-04T16:11:57.270303+0000","last_fullsized":"2024-03-04T16:11:57.270303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:40.919972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:40.919972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"90'13","reported_seq":410,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.279644+0000","last_change":"2024-03-04T16:10:29.383816+0000","last_active":"2024-03-04T16:11:57.279644+0000","last_peered":"2024-03-04T16:11:57.279644+0000","last_clean":"2024-03-04T16:11:57.279644+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:11:57.279644+0000","last_undegraded":"2024-03-04T16:11:57.279644+0000","last_fullsized":"2024-03-04T16:11:57.279644+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":344,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.332548+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:11:50.332548+0000","last_peered":"2024-03-04T16:11:50.332548+0000","last_clean":"2024-03-04T16:11:50.332548+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:11:50.332548+0000","last_undegraded":"2024-03-04T16:11:50.332548+0000","last_fullsized":"2024-03-04T16:11:50.332548+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":354,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.279474+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:11:57.279474+0000","last_peered":"2024-03-04T16:11:57.279474+0000","last_clean":"2024-03-04T16:11:57.279474+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:11:57.279474+0000","last_undegraded":"2024-03-04T16:11:57.279474+0000","last_fullsized":"2024-03-04T16:11:57.279474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":368,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268713+0000","last_change":"2024-03-04T16:10:29.247074+0000","last_active":"2024-03-04T16:11:57.268713+0000","last_peered":"2024-03-04T16:11:57.268713+0000","last_clean":"2024-03-04T16:11:57.268713+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:11:57.268713+0000","last_undegraded":"2024-03-04T16:11:57.268713+0000","last_fullsized":"2024-03-04T16:11:57.268713+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.796905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"75'11","reported_seq":395,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268377+0000","last_change":"2024-03-04T16:10:29.372889+0000","last_active":"2024-03-04T16:11:57.268377+0000","last_peered":"2024-03-04T16:11:57.268377+0000","last_clean":"2024-03-04T16:11:57.268377+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:11:57.268377+0000","last_undegraded":"2024-03-04T16:11:57.268377+0000","last_fullsized":"2024-03-04T16:11:57.268377+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":224,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270263+0000","last_change":"2024-03-04T16:09:11.858837+0000","last_active":"2024-03-04T16:11:57.270263+0000","last_peered":"2024-03-04T16:11:57.270263+0000","last_clean":"2024-03-04T16:11:57.270263+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:11:57.270263+0000","last_undegraded":"2024-03-04T16:11:57.270263+0000","last_fullsized":"2024-03-04T16:11:57.270263+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:11.858712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:11.858712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":486,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274002+0000","last_change":"2024-03-04T16:08:23.208622+0000","last_active":"2024-03-04T16:11:57.274002+0000","last_peered":"2024-03-04T16:11:57.274002+0000","last_clean":"2024-03-04T16:11:57.274002+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:11:57.274002+0000","last_undegraded":"2024-03-04T16:11:57.274002+0000","last_fullsized":"2024-03-04T16:11:57.274002+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:08:23.208503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:23.208503+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488390+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":396,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269763+0000","last_change":"2024-03-04T16:11:05.876940+0000","last_active":"2024-03-04T16:11:57.269763+0000","last_peered":"2024-03-04T16:11:57.269763+0000","last_clean":"2024-03-04T16:11:57.269763+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:11:57.269763+0000","last_undegraded":"2024-03-04T16:11:57.269763+0000","last_fullsized":"2024-03-04T16:11:57.269763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:11:05.876809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:11:05.876809+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:10.772164+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"75'10","reported_seq":391,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.334003+0000","last_change":"2024-03-04T16:10:17.514015+0000","last_active":"2024-03-04T16:11:50.334003+0000","last_peered":"2024-03-04T16:11:50.334003+0000","last_clean":"2024-03-04T16:11:50.334003+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:11:50.334003+0000","last_undegraded":"2024-03-04T16:11:50.334003+0000","last_fullsized":"2024-03-04T16:11:50.334003+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":348,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269125+0000","last_change":"2024-03-04T16:10:17.464338+0000","last_active":"2024-03-04T16:11:57.269125+0000","last_peered":"2024-03-04T16:11:57.269125+0000","last_clean":"2024-03-04T16:11:57.269125+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:11:57.269125+0000","last_undegraded":"2024-03-04T16:11:57.269125+0000","last_fullsized":"2024-03-04T16:11:57.269125+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.876691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.876691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":465,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.334113+0000","last_change":"2024-03-04T16:10:17.498945+0000","last_active":"2024-03-04T16:11:50.334113+0000","last_peered":"2024-03-04T16:11:50.334113+0000","last_clean":"2024-03-04T16:11:50.334113+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:11:50.334113+0000","last_undegraded":"2024-03-04T16:11:50.334113+0000","last_fullsized":"2024-03-04T16:11:50.334113+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-04T16:08:07.413781+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:08:07.413781+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":256,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269593+0000","last_change":"2024-03-04T16:10:50.000077+0000","last_active":"2024-03-04T16:11:57.269593+0000","last_peered":"2024-03-04T16:11:57.269593+0000","last_clean":"2024-03-04T16:11:57.269593+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:11:57.269593+0000","last_undegraded":"2024-03-04T16:11:57.269593+0000","last_fullsized":"2024-03-04T16:11:57.269593+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:10:49.999919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:49.999919+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.077817+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"75'9","reported_seq":373,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274249+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:11:57.274249+0000","last_peered":"2024-03-04T16:11:57.274249+0000","last_clean":"2024-03-04T16:11:57.274249+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:11:57.274249+0000","last_undegraded":"2024-03-04T16:11:57.274249+0000","last_fullsized":"2024-03-04T16:11:57.274249+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":82,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268419+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:11:57.268419+0000","last_peered":"2024-03-04T16:11:57.268419+0000","last_clean":"2024-03-04T16:11:57.268419+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:11:57.268419+0000","last_undegraded":"2024-03-04T16:11:57.268419+0000","last_fullsized":"2024-03-04T16:11:57.268419+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":344,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.335277+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:11:50.335277+0000","last_peered":"2024-03-04T16:11:50.335277+0000","last_clean":"2024-03-04T16:11:50.335277+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:11:50.335277+0000","last_undegraded":"2024-03-04T16:11:50.335277+0000","last_fullsized":"2024-03-04T16:11:50.335277+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269423+0000","last_change":"2024-03-04T16:10:29.243085+0000","last_active":"2024-03-04T16:11:57.269423+0000","last_peered":"2024-03-04T16:11:57.269423+0000","last_clean":"2024-03-04T16:11:57.269423+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:11:57.269423+0000","last_undegraded":"2024-03-04T16:11:57.269423+0000","last_fullsized":"2024-03-04T16:11:57.269423+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.827513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.827513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"75'9","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274879+0000","last_change":"2024-03-04T16:10:09.625107+0000","last_active":"2024-03-04T16:11:57.274879+0000","last_peered":"2024-03-04T16:11:57.274879+0000","last_clean":"2024-03-04T16:11:57.274879+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:11:57.274879+0000","last_undegraded":"2024-03-04T16:11:57.274879+0000","last_fullsized":"2024-03-04T16:11:57.274879+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:09.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.624847+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361297+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270195+0000","last_change":"2024-03-04T16:10:29.370895+0000","last_active":"2024-03-04T16:11:57.270195+0000","last_peered":"2024-03-04T16:11:57.270195+0000","last_clean":"2024-03-04T16:11:57.270195+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:11:57.270195+0000","last_undegraded":"2024-03-04T16:11:57.270195+0000","last_fullsized":"2024-03-04T16:11:57.270195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.797282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":353,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273701+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:11:57.273701+0000","last_peered":"2024-03-04T16:11:57.273701+0000","last_clean":"2024-03-04T16:11:57.273701+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:11:57.273701+0000","last_undegraded":"2024-03-04T16:11:57.273701+0000","last_fullsized":"2024-03-04T16:11:57.273701+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488685+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":260,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275159+0000","last_change":"2024-03-04T16:10:29.242936+0000","last_active":"2024-03-04T16:11:57.275159+0000","last_peered":"2024-03-04T16:11:57.275159+0000","last_clean":"2024-03-04T16:11:57.275159+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:11:57.275159+0000","last_undegraded":"2024-03-04T16:11:57.275159+0000","last_fullsized":"2024-03-04T16:11:57.275159+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.906439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.906439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361263+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"86'9","reported_seq":360,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.335493+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:11:50.335493+0000","last_peered":"2024-03-04T16:11:50.335493+0000","last_clean":"2024-03-04T16:11:50.335493+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:11:50.335493+0000","last_undegraded":"2024-03-04T16:11:50.335493+0000","last_fullsized":"2024-03-04T16:11:50.335493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"226'12166","reported_seq":25176,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.535737+0000","last_change":"2024-03-04T16:11:22.402910+0000","last_active":"2024-03-04T16:11:57.535737+0000","last_peered":"2024-03-04T16:11:57.535737+0000","last_clean":"2024-03-04T16:11:57.535737+0000","last_became_active":"2024-03-04T16:11:22.387111+0000","last_became_peered":"2024-03-04T16:11:22.387111+0000","last_unstale":"2024-03-04T16:11:57.535737+0000","last_undegraded":"2024-03-04T16:11:57.535737+0000","last_fullsized":"2024-03-04T16:11:57.535737+0000","mapping_epoch":213,"log_start":"199'7631","ondisk_log_start":"199'7631","created":167,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'5174","last_scrub_stamp":"2024-03-04T16:09:57.354140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.354140+0000","objects_scrubbed":5128,"log_size":4535,"log_dups_size":0,"ondisk_log_size":4535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.490150+0000","scrub_duration":1281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185472,"num_objects":2902,"num_object_clones":0,"num_object_copies":8706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12166,"num_write_kb":482176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6399,"num_bytes_recovered":419364864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":356,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275568+0000","last_change":"2024-03-04T16:10:29.344188+0000","last_active":"2024-03-04T16:11:57.275568+0000","last_peered":"2024-03-04T16:11:57.275568+0000","last_clean":"2024-03-04T16:11:57.275568+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:11:57.275568+0000","last_undegraded":"2024-03-04T16:11:57.275568+0000","last_fullsized":"2024-03-04T16:11:57.275568+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488632+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":399,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270484+0000","last_change":"2024-03-04T16:10:09.721121+0000","last_active":"2024-03-04T16:11:57.270484+0000","last_peered":"2024-03-04T16:11:57.270484+0000","last_clean":"2024-03-04T16:11:57.270484+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:11:57.270484+0000","last_undegraded":"2024-03-04T16:11:57.270484+0000","last_fullsized":"2024-03-04T16:11:57.270484+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:10:09.720996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.720996+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"75'15","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274619+0000","last_change":"2024-03-04T16:10:29.370436+0000","last_active":"2024-03-04T16:11:57.274619+0000","last_peered":"2024-03-04T16:11:57.274619+0000","last_clean":"2024-03-04T16:11:57.274619+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:11:57.274619+0000","last_undegraded":"2024-03-04T16:11:57.274619+0000","last_fullsized":"2024-03-04T16:11:57.274619+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:26.222082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.222082+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488361+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":484,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269635+0000","last_change":"2024-03-04T16:10:39.929934+0000","last_active":"2024-03-04T16:11:57.269635+0000","last_peered":"2024-03-04T16:11:57.269635+0000","last_clean":"2024-03-04T16:11:57.269635+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:11:57.269635+0000","last_undegraded":"2024-03-04T16:11:57.269635+0000","last_fullsized":"2024-03-04T16:11:57.269635+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:10:39.929809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:39.929809+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"226'12509","reported_seq":26228,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.481855+0000","last_change":"2024-03-04T16:11:17.323969+0000","last_active":"2024-03-04T16:11:57.481855+0000","last_peered":"2024-03-04T16:11:57.481855+0000","last_clean":"2024-03-04T16:11:57.481855+0000","last_became_active":"2024-03-04T16:11:17.309642+0000","last_became_peered":"2024-03-04T16:11:17.309642+0000","last_unstale":"2024-03-04T16:11:57.481855+0000","last_undegraded":"2024-03-04T16:11:57.481855+0000","last_fullsized":"2024-03-04T16:11:57.481855+0000","mapping_epoch":209,"log_start":"199'8271","ondisk_log_start":"199'8271","created":167,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196149248,"num_objects":2993,"num_object_clones":0,"num_object_copies":8979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12509,"num_write_kb":496064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6788,"num_bytes_recovered":444858368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":333,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333820+0000","last_change":"2024-03-04T16:08:13.509602+0000","last_active":"2024-03-04T16:11:50.333820+0000","last_peered":"2024-03-04T16:11:50.333820+0000","last_clean":"2024-03-04T16:11:50.333820+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:11:50.333820+0000","last_undegraded":"2024-03-04T16:11:50.333820+0000","last_fullsized":"2024-03-04T16:11:50.333820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.509441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.509441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":363,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270385+0000","last_change":"2024-03-04T16:09:58.574092+0000","last_active":"2024-03-04T16:11:57.270385+0000","last_peered":"2024-03-04T16:11:57.270385+0000","last_clean":"2024-03-04T16:11:57.270385+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:11:57.270385+0000","last_undegraded":"2024-03-04T16:11:57.270385+0000","last_fullsized":"2024-03-04T16:11:57.270385+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:58.573650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.573650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"75'12","reported_seq":382,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267624+0000","last_change":"2024-03-04T16:09:57.809587+0000","last_active":"2024-03-04T16:11:57.267624+0000","last_peered":"2024-03-04T16:11:57.267624+0000","last_clean":"2024-03-04T16:11:57.267624+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:11:57.267624+0000","last_undegraded":"2024-03-04T16:11:57.267624+0000","last_fullsized":"2024-03-04T16:11:57.267624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:09:57.809460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.809460+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":252,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274758+0000","last_change":"2024-03-04T16:10:17.478101+0000","last_active":"2024-03-04T16:11:57.274758+0000","last_peered":"2024-03-04T16:11:57.274758+0000","last_clean":"2024-03-04T16:11:57.274758+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:11:57.274758+0000","last_undegraded":"2024-03-04T16:11:57.274758+0000","last_fullsized":"2024-03-04T16:11:57.274758+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:18.939310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:18.939310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361083+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.6","version":"226'12219","reported_seq":25099,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:58.147420+0000","last_change":"2024-03-04T16:11:27.265041+0000","last_active":"2024-03-04T16:11:58.147420+0000","last_peered":"2024-03-04T16:11:58.147420+0000","last_clean":"2024-03-04T16:11:58.147420+0000","last_became_active":"2024-03-04T16:11:27.250184+0000","last_became_peered":"2024-03-04T16:11:27.250184+0000","last_unstale":"2024-03-04T16:11:58.147420+0000","last_undegraded":"2024-03-04T16:11:58.147420+0000","last_fullsized":"2024-03-04T16:11:58.147420+0000","mapping_epoch":215,"log_start":"199'8180","ondisk_log_start":"199'8180","created":167,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4039,"log_dups_size":0,"ondisk_log_size":4039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192086016,"num_objects":2931,"num_object_clones":0,"num_object_copies":8793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12219,"num_write_kb":484800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6366,"num_bytes_recovered":417202176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269217+0000","last_change":"2024-03-04T16:10:29.248916+0000","last_active":"2024-03-04T16:11:57.269217+0000","last_peered":"2024-03-04T16:11:57.269217+0000","last_clean":"2024-03-04T16:11:57.269217+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:11:57.269217+0000","last_undegraded":"2024-03-04T16:11:57.269217+0000","last_fullsized":"2024-03-04T16:11:57.269217+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:10.829985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:10.829985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":82,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275556+0000","last_change":"2024-03-04T16:10:17.466540+0000","last_active":"2024-03-04T16:11:57.275556+0000","last_peered":"2024-03-04T16:11:57.275556+0000","last_clean":"2024-03-04T16:11:57.275556+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:11:57.275556+0000","last_undegraded":"2024-03-04T16:11:57.275556+0000","last_fullsized":"2024-03-04T16:11:57.275556+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.936788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.936788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":369,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270511+0000","last_change":"2024-03-04T16:10:29.371684+0000","last_active":"2024-03-04T16:11:57.270511+0000","last_peered":"2024-03-04T16:11:57.270511+0000","last_clean":"2024-03-04T16:11:57.270511+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:11:57.270511+0000","last_undegraded":"2024-03-04T16:11:57.270511+0000","last_fullsized":"2024-03-04T16:11:57.270511+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:57.610425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.610425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"75'19","reported_seq":144,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.279330+0000","last_change":"2024-03-04T16:10:29.392036+0000","last_active":"2024-03-04T16:11:57.279330+0000","last_peered":"2024-03-04T16:11:57.279330+0000","last_clean":"2024-03-04T16:11:57.279330+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:11:57.279330+0000","last_undegraded":"2024-03-04T16:11:57.279330+0000","last_fullsized":"2024-03-04T16:11:57.279330+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"226'12221","reported_seq":25148,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:58.024918+0000","last_change":"2024-03-04T16:11:21.402480+0000","last_active":"2024-03-04T16:11:58.024918+0000","last_peered":"2024-03-04T16:11:58.024918+0000","last_clean":"2024-03-04T16:11:58.024918+0000","last_became_active":"2024-03-04T16:11:21.384637+0000","last_became_peered":"2024-03-04T16:11:21.384637+0000","last_unstale":"2024-03-04T16:11:58.024918+0000","last_undegraded":"2024-03-04T16:11:58.024918+0000","last_fullsized":"2024-03-04T16:11:58.024918+0000","mapping_epoch":212,"log_start":"200'8503","ondisk_log_start":"200'8503","created":167,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3718,"log_dups_size":0,"ondisk_log_size":3718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183959552,"num_objects":2807,"num_object_clones":0,"num_object_copies":8421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12221,"num_write_kb":480896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6391,"num_bytes_recovered":418840576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":238,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269245+0000","last_change":"2024-03-04T16:10:47.029618+0000","last_active":"2024-03-04T16:11:57.269245+0000","last_peered":"2024-03-04T16:11:57.269245+0000","last_clean":"2024-03-04T16:11:57.269245+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:11:57.269245+0000","last_undegraded":"2024-03-04T16:11:57.269245+0000","last_fullsized":"2024-03-04T16:11:57.269245+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:47.029491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:10:47.029491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:10.136972+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":82,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268305+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:11:57.268305+0000","last_peered":"2024-03-04T16:11:57.268305+0000","last_clean":"2024-03-04T16:11:57.268305+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:11:57.268305+0000","last_undegraded":"2024-03-04T16:11:57.268305+0000","last_fullsized":"2024-03-04T16:11:57.268305+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":368,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275591+0000","last_change":"2024-03-04T16:10:29.354903+0000","last_active":"2024-03-04T16:11:57.275591+0000","last_peered":"2024-03-04T16:11:57.275591+0000","last_clean":"2024-03-04T16:11:57.275591+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:11:57.275591+0000","last_undegraded":"2024-03-04T16:11:57.275591+0000","last_fullsized":"2024-03-04T16:11:57.275591+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"114'13","reported_seq":208,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275399+0000","last_change":"2024-03-04T16:10:12.706114+0000","last_active":"2024-03-04T16:11:57.275399+0000","last_peered":"2024-03-04T16:11:57.275399+0000","last_clean":"2024-03-04T16:11:57.275399+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:11:57.275399+0000","last_undegraded":"2024-03-04T16:11:57.275399+0000","last_fullsized":"2024-03-04T16:11:57.275399+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:10:12.705971+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:10:12.705971+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361397+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"226'12358","reported_seq":12485,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.538728+0000","last_change":"2024-03-04T16:08:56.789743+0000","last_active":"2024-03-04T16:11:57.538728+0000","last_peered":"2024-03-04T16:11:57.538728+0000","last_clean":"2024-03-04T16:11:57.538728+0000","last_became_active":"2024-03-04T16:08:56.789006+0000","last_became_peered":"2024-03-04T16:08:56.789006+0000","last_unstale":"2024-03-04T16:11:57.538728+0000","last_undegraded":"2024-03-04T16:11:57.538728+0000","last_fullsized":"2024-03-04T16:11:57.538728+0000","mapping_epoch":167,"log_start":"199'7863","ondisk_log_start":"199'7863","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195690496,"num_objects":2986,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12358,"num_write_kb":491008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":341,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.334593+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:11:50.334593+0000","last_peered":"2024-03-04T16:11:50.334593+0000","last_clean":"2024-03-04T16:11:50.334593+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:11:50.334593+0000","last_undegraded":"2024-03-04T16:11:50.334593+0000","last_fullsized":"2024-03-04T16:11:50.334593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":383,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275245+0000","last_change":"2024-03-04T16:10:29.374989+0000","last_active":"2024-03-04T16:11:57.275245+0000","last_peered":"2024-03-04T16:11:57.275245+0000","last_clean":"2024-03-04T16:11:57.275245+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:11:57.275245+0000","last_undegraded":"2024-03-04T16:11:57.275245+0000","last_fullsized":"2024-03-04T16:11:57.275245+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":381,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267968+0000","last_change":"2024-03-04T16:10:29.321793+0000","last_active":"2024-03-04T16:11:57.267968+0000","last_peered":"2024-03-04T16:11:57.267968+0000","last_clean":"2024-03-04T16:11:57.267968+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:11:57.267968+0000","last_undegraded":"2024-03-04T16:11:57.267968+0000","last_fullsized":"2024-03-04T16:11:57.267968+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:58.764814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.764814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"195'30","reported_seq":566,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278775+0000","last_change":"2024-03-04T16:10:29.319330+0000","last_active":"2024-03-04T16:11:57.278775+0000","last_peered":"2024-03-04T16:11:57.278775+0000","last_clean":"2024-03-04T16:11:57.278775+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:11:57.278775+0000","last_undegraded":"2024-03-04T16:11:57.278775+0000","last_fullsized":"2024-03-04T16:11:57.278775+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'28","last_scrub_stamp":"2024-03-04T16:08:21.818513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:21.818513+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":191,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274401+0000","last_change":"2024-03-04T16:08:55.084564+0000","last_active":"2024-03-04T16:11:57.274401+0000","last_peered":"2024-03-04T16:11:57.274401+0000","last_clean":"2024-03-04T16:11:57.274401+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:11:57.274401+0000","last_undegraded":"2024-03-04T16:11:57.274401+0000","last_fullsized":"2024-03-04T16:11:57.274401+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.084487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.084487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"225'12181","reported_seq":12308,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:53.386474+0000","last_change":"2024-03-04T16:08:56.231801+0000","last_active":"2024-03-04T16:11:53.386474+0000","last_peered":"2024-03-04T16:11:53.386474+0000","last_clean":"2024-03-04T16:11:53.386474+0000","last_became_active":"2024-03-04T16:08:56.231512+0000","last_became_peered":"2024-03-04T16:08:56.231512+0000","last_unstale":"2024-03-04T16:11:53.386474+0000","last_undegraded":"2024-03-04T16:11:53.386474+0000","last_fullsized":"2024-03-04T16:11:53.386474+0000","mapping_epoch":167,"log_start":"199'8210","ondisk_log_start":"199'8210","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215023616,"num_objects":3281,"num_object_clones":0,"num_object_copies":9843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3281,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":494784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":357,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270645+0000","last_change":"2024-03-04T16:10:44.996119+0000","last_active":"2024-03-04T16:11:57.270645+0000","last_peered":"2024-03-04T16:11:57.270645+0000","last_clean":"2024-03-04T16:11:57.270645+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:11:57.270645+0000","last_undegraded":"2024-03-04T16:11:57.270645+0000","last_fullsized":"2024-03-04T16:11:57.270645+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:44.995993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:44.995993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:07.188647+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":375,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278692+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:11:57.278692+0000","last_peered":"2024-03-04T16:11:57.278692+0000","last_clean":"2024-03-04T16:11:57.278692+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:11:57.278692+0000","last_undegraded":"2024-03-04T16:11:57.278692+0000","last_fullsized":"2024-03-04T16:11:57.278692+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.332560+0000","last_change":"2024-03-04T16:10:17.480211+0000","last_active":"2024-03-04T16:11:50.332560+0000","last_peered":"2024-03-04T16:11:50.332560+0000","last_clean":"2024-03-04T16:11:50.332560+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:11:50.332560+0000","last_undegraded":"2024-03-04T16:11:50.332560+0000","last_fullsized":"2024-03-04T16:11:50.332560+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"90'18","reported_seq":37,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270192+0000","last_change":"2024-03-04T16:11:38.589156+0000","last_active":"2024-03-04T16:11:57.270192+0000","last_peered":"2024-03-04T16:11:57.270192+0000","last_clean":"2024-03-04T16:11:57.270192+0000","last_became_active":"2024-03-04T16:11:38.576380+0000","last_became_peered":"2024-03-04T16:11:38.576380+0000","last_unstale":"2024-03-04T16:11:57.270192+0000","last_undegraded":"2024-03-04T16:11:57.270192+0000","last_fullsized":"2024-03-04T16:11:57.270192+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:08:26.885716+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:08:26.885716+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":82,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273924+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:11:57.273924+0000","last_peered":"2024-03-04T16:11:57.273924+0000","last_clean":"2024-03-04T16:11:57.273924+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:11:57.273924+0000","last_undegraded":"2024-03-04T16:11:57.273924+0000","last_fullsized":"2024-03-04T16:11:57.273924+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"226'12051","reported_seq":21954,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.535134+0000","last_change":"2024-03-04T16:11:48.294302+0000","last_active":"2024-03-04T16:11:57.535134+0000","last_peered":"2024-03-04T16:11:57.535134+0000","last_clean":"2024-03-04T16:11:57.535134+0000","last_became_active":"2024-03-04T16:11:48.276247+0000","last_became_peered":"2024-03-04T16:11:48.276247+0000","last_unstale":"2024-03-04T16:11:57.535134+0000","last_undegraded":"2024-03-04T16:11:57.535134+0000","last_fullsized":"2024-03-04T16:11:57.535134+0000","mapping_epoch":222,"log_start":"199'7501","ondisk_log_start":"199'7501","created":167,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196673536,"num_objects":3001,"num_object_clones":0,"num_object_copies":9003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":481664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4881,"num_bytes_recovered":319881216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":350,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273199+0000","last_change":"2024-03-04T16:09:07.122337+0000","last_active":"2024-03-04T16:11:57.273199+0000","last_peered":"2024-03-04T16:11:57.273199+0000","last_clean":"2024-03-04T16:11:57.273199+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:11:57.273199+0000","last_undegraded":"2024-03-04T16:11:57.273199+0000","last_fullsized":"2024-03-04T16:11:57.273199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:07.122197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:07.122197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488550+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":669,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269332+0000","last_change":"2024-03-04T16:10:33.829546+0000","last_active":"2024-03-04T16:11:57.269332+0000","last_peered":"2024-03-04T16:11:57.269332+0000","last_clean":"2024-03-04T16:11:57.269332+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:11:57.269332+0000","last_undegraded":"2024-03-04T16:11:57.269332+0000","last_fullsized":"2024-03-04T16:11:57.269332+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:10:33.829403+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:33.829403+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":359,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.332808+0000","last_change":"2024-03-04T16:10:29.340288+0000","last_active":"2024-03-04T16:11:50.332808+0000","last_peered":"2024-03-04T16:11:50.332808+0000","last_clean":"2024-03-04T16:11:50.332808+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:11:50.332808+0000","last_undegraded":"2024-03-04T16:11:50.332808+0000","last_fullsized":"2024-03-04T16:11:50.332808+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"195'17","reported_seq":153,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268567+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:11:57.268567+0000","last_peered":"2024-03-04T16:11:57.268567+0000","last_clean":"2024-03-04T16:11:57.268567+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:11:57.268567+0000","last_undegraded":"2024-03-04T16:11:57.268567+0000","last_fullsized":"2024-03-04T16:11:57.268567+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269944+0000","last_change":"2024-03-04T16:10:17.465880+0000","last_active":"2024-03-04T16:11:57.269944+0000","last_peered":"2024-03-04T16:11:57.269944+0000","last_clean":"2024-03-04T16:11:57.269944+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:11:57.269944+0000","last_undegraded":"2024-03-04T16:11:57.269944+0000","last_fullsized":"2024-03-04T16:11:57.269944+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.788580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.788580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"226'12247","reported_seq":26058,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.988303+0000","last_change":"2024-03-04T16:11:17.338038+0000","last_active":"2024-03-04T16:11:57.988303+0000","last_peered":"2024-03-04T16:11:57.988303+0000","last_clean":"2024-03-04T16:11:57.988303+0000","last_became_active":"2024-03-04T16:11:17.319156+0000","last_became_peered":"2024-03-04T16:11:17.319156+0000","last_unstale":"2024-03-04T16:11:57.988303+0000","last_undegraded":"2024-03-04T16:11:57.988303+0000","last_fullsized":"2024-03-04T16:11:57.988303+0000","mapping_epoch":209,"log_start":"200'8538","ondisk_log_start":"200'8538","created":167,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":8787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12247,"num_write_kb":485632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6834,"num_bytes_recovered":447873024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":351,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278385+0000","last_change":"2024-03-04T16:08:28.885544+0000","last_active":"2024-03-04T16:11:57.278385+0000","last_peered":"2024-03-04T16:11:57.278385+0000","last_clean":"2024-03-04T16:11:57.278385+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:11:57.278385+0000","last_undegraded":"2024-03-04T16:11:57.278385+0000","last_fullsized":"2024-03-04T16:11:57.278385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:28.885438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:08:28.885438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":199,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.276499+0000","last_change":"2024-03-04T16:10:11.674758+0000","last_active":"2024-03-04T16:11:57.276499+0000","last_peered":"2024-03-04T16:11:57.276499+0000","last_clean":"2024-03-04T16:11:57.276499+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:11:57.276499+0000","last_undegraded":"2024-03-04T16:11:57.276499+0000","last_fullsized":"2024-03-04T16:11:57.276499+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.674483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.674483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361234+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":931,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269245+0000","last_change":"2024-03-04T16:10:59.956106+0000","last_active":"2024-03-04T16:11:57.269245+0000","last_peered":"2024-03-04T16:11:57.269245+0000","last_clean":"2024-03-04T16:11:57.269245+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:11:57.269245+0000","last_undegraded":"2024-03-04T16:11:57.269245+0000","last_fullsized":"2024-03-04T16:11:57.269245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:10:59.955973+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:10:59.955973+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:20.849731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":547,"num_read_kb":542,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"75'12","reported_seq":233,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274635+0000","last_change":"2024-03-04T16:10:17.517202+0000","last_active":"2024-03-04T16:11:57.274635+0000","last_peered":"2024-03-04T16:11:57.274635+0000","last_clean":"2024-03-04T16:11:57.274635+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:11:57.274635+0000","last_undegraded":"2024-03-04T16:11:57.274635+0000","last_fullsized":"2024-03-04T16:11:57.274635+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:10:00.540297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.540297+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361362+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273608+0000","last_change":"2024-03-04T16:09:55.100999+0000","last_active":"2024-03-04T16:11:57.273608+0000","last_peered":"2024-03-04T16:11:57.273608+0000","last_clean":"2024-03-04T16:11:57.273608+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:11:57.273608+0000","last_undegraded":"2024-03-04T16:11:57.273608+0000","last_fullsized":"2024-03-04T16:11:57.273608+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.100871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.100871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488892+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"225'11701","reported_seq":25509,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:53.394435+0000","last_change":"2024-03-04T16:11:11.358039+0000","last_active":"2024-03-04T16:11:53.394435+0000","last_peered":"2024-03-04T16:11:53.394435+0000","last_clean":"2024-03-04T16:11:53.394435+0000","last_became_active":"2024-03-04T16:11:11.278473+0000","last_became_peered":"2024-03-04T16:11:11.278473+0000","last_unstale":"2024-03-04T16:11:53.394435+0000","last_undegraded":"2024-03-04T16:11:53.394435+0000","last_fullsized":"2024-03-04T16:11:53.394435+0000","mapping_epoch":205,"log_start":"199'7775","ondisk_log_start":"199'7775","created":167,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211222528,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11701,"num_write_kb":477568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6831,"num_bytes_recovered":447610880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":363,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.281154+0000","last_change":"2024-03-04T16:10:29.247615+0000","last_active":"2024-03-04T16:11:57.281154+0000","last_peered":"2024-03-04T16:11:57.281154+0000","last_clean":"2024-03-04T16:11:57.281154+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:11:57.281154+0000","last_undegraded":"2024-03-04T16:11:57.281154+0000","last_fullsized":"2024-03-04T16:11:57.281154+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.995029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.995029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278637+0000","last_change":"2024-03-04T16:10:17.491352+0000","last_active":"2024-03-04T16:11:57.278637+0000","last_peered":"2024-03-04T16:11:57.278637+0000","last_clean":"2024-03-04T16:11:57.278637+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:11:57.278637+0000","last_undegraded":"2024-03-04T16:11:57.278637+0000","last_fullsized":"2024-03-04T16:11:57.278637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"159'27","reported_seq":322,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273005+0000","last_change":"2024-03-04T16:08:24.229850+0000","last_active":"2024-03-04T16:11:57.273005+0000","last_peered":"2024-03-04T16:11:57.273005+0000","last_clean":"2024-03-04T16:11:57.273005+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:11:57.273005+0000","last_undegraded":"2024-03-04T16:11:57.273005+0000","last_fullsized":"2024-03-04T16:11:57.273005+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.229746+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488468+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275351+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:11:57.275351+0000","last_peered":"2024-03-04T16:11:57.275351+0000","last_clean":"2024-03-04T16:11:57.275351+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:11:57.275351+0000","last_undegraded":"2024-03-04T16:11:57.275351+0000","last_fullsized":"2024-03-04T16:11:57.275351+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.490185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"226'12385","reported_seq":21861,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:58.155101+0000","last_change":"2024-03-04T16:11:50.346372+0000","last_active":"2024-03-04T16:11:58.155101+0000","last_peered":"2024-03-04T16:11:58.155101+0000","last_clean":"2024-03-04T16:11:58.155101+0000","last_became_active":"2024-03-04T16:11:50.303254+0000","last_became_peered":"2024-03-04T16:11:50.303254+0000","last_unstale":"2024-03-04T16:11:58.155101+0000","last_undegraded":"2024-03-04T16:11:58.155101+0000","last_fullsized":"2024-03-04T16:11:58.155101+0000","mapping_epoch":224,"log_start":"200'8384","ondisk_log_start":"200'8384","created":167,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190906368,"num_objects":2913,"num_object_clones":0,"num_object_copies":8739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12385,"num_write_kb":489536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4659,"num_bytes_recovered":305266688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":357,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.275059+0000","last_change":"2024-03-04T16:10:17.467416+0000","last_active":"2024-03-04T16:11:57.275059+0000","last_peered":"2024-03-04T16:11:57.275059+0000","last_clean":"2024-03-04T16:11:57.275059+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:11:57.275059+0000","last_undegraded":"2024-03-04T16:11:57.275059+0000","last_fullsized":"2024-03-04T16:11:57.275059+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.314502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.314502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361171+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":374,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274103+0000","last_change":"2024-03-04T16:10:29.368316+0000","last_active":"2024-03-04T16:11:57.274103+0000","last_peered":"2024-03-04T16:11:57.274103+0000","last_clean":"2024-03-04T16:11:57.274103+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:11:57.274103+0000","last_undegraded":"2024-03-04T16:11:57.274103+0000","last_fullsized":"2024-03-04T16:11:57.274103+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"115'15","reported_seq":379,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.279554+0000","last_change":"2024-03-04T16:10:01.014399+0000","last_active":"2024-03-04T16:11:57.279554+0000","last_peered":"2024-03-04T16:11:57.279554+0000","last_clean":"2024-03-04T16:11:57.279554+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:11:57.279554+0000","last_undegraded":"2024-03-04T16:11:57.279554+0000","last_fullsized":"2024-03-04T16:11:57.279554+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:10:01.014225+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.014225+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":121,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.272848+0000","last_change":"2024-03-04T16:10:29.369889+0000","last_active":"2024-03-04T16:11:57.272848+0000","last_peered":"2024-03-04T16:11:57.272848+0000","last_clean":"2024-03-04T16:11:57.272848+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:11:57.272848+0000","last_undegraded":"2024-03-04T16:11:57.272848+0000","last_fullsized":"2024-03-04T16:11:57.272848+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:10:03.917637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.917637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.d","version":"226'12305","reported_seq":29086,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.520642+0000","last_change":"2024-03-04T16:11:20.378691+0000","last_active":"2024-03-04T16:11:57.520642+0000","last_peered":"2024-03-04T16:11:57.520642+0000","last_clean":"2024-03-04T16:11:57.520642+0000","last_became_active":"2024-03-04T16:11:20.363111+0000","last_became_peered":"2024-03-04T16:11:20.363111+0000","last_unstale":"2024-03-04T16:11:57.520642+0000","last_undegraded":"2024-03-04T16:11:57.520642+0000","last_fullsized":"2024-03-04T16:11:57.520642+0000","mapping_epoch":211,"log_start":"199'7804","ondisk_log_start":"199'7804","created":167,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190185496,"num_objects":2903,"num_object_clones":0,"num_object_copies":8709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12305,"num_write_kb":486593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8312,"num_bytes_recovered":544669720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":363,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.269944+0000","last_change":"2024-03-04T16:10:29.245334+0000","last_active":"2024-03-04T16:11:57.269944+0000","last_peered":"2024-03-04T16:11:57.269944+0000","last_clean":"2024-03-04T16:11:57.269944+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:11:57.269944+0000","last_undegraded":"2024-03-04T16:11:57.269944+0000","last_fullsized":"2024-03-04T16:11:57.269944+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:14.851027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:09:14.851027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":203,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.274732+0000","last_change":"2024-03-04T16:09:24.277958+0000","last_active":"2024-03-04T16:11:57.274732+0000","last_peered":"2024-03-04T16:11:57.274732+0000","last_clean":"2024-03-04T16:11:57.274732+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:11:57.274732+0000","last_undegraded":"2024-03-04T16:11:57.274732+0000","last_fullsized":"2024-03-04T16:11:57.274732+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:09:24.277674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:24.277674+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.361205+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"90'21","reported_seq":401,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267566+0000","last_change":"2024-03-04T16:10:29.368068+0000","last_active":"2024-03-04T16:11:57.267566+0000","last_peered":"2024-03-04T16:11:57.267566+0000","last_clean":"2024-03-04T16:11:57.267566+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:11:57.267566+0000","last_undegraded":"2024-03-04T16:11:57.267566+0000","last_fullsized":"2024-03-04T16:11:57.267566+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:08:59.783838+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:08:59.783838+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":463,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333133+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:11:50.333133+0000","last_peered":"2024-03-04T16:11:50.333133+0000","last_clean":"2024-03-04T16:11:50.333133+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:11:50.333133+0000","last_undegraded":"2024-03-04T16:11:50.333133+0000","last_fullsized":"2024-03-04T16:11:50.333133+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"226'12606","reported_seq":12735,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:58.139927+0000","last_change":"2024-03-04T16:08:56.788817+0000","last_active":"2024-03-04T16:11:58.139927+0000","last_peered":"2024-03-04T16:11:58.139927+0000","last_clean":"2024-03-04T16:11:58.139927+0000","last_became_active":"2024-03-04T16:08:56.788513+0000","last_became_peered":"2024-03-04T16:08:56.788513+0000","last_unstale":"2024-03-04T16:11:58.139927+0000","last_undegraded":"2024-03-04T16:11:58.139927+0000","last_fullsized":"2024-03-04T16:11:58.139927+0000","mapping_epoch":167,"log_start":"200'8596","ondisk_log_start":"200'8596","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4010,"log_dups_size":0,"ondisk_log_size":4010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:11:58.360803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191234048,"num_objects":2918,"num_object_clones":0,"num_object_copies":8754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267978+0000","last_change":"2024-03-04T16:10:29.247041+0000","last_active":"2024-03-04T16:11:57.267978+0000","last_peered":"2024-03-04T16:11:57.267978+0000","last_clean":"2024-03-04T16:11:57.267978+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:11:57.267978+0000","last_undegraded":"2024-03-04T16:11:57.267978+0000","last_fullsized":"2024-03-04T16:11:57.267978+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":365,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.335075+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:11:50.335075+0000","last_peered":"2024-03-04T16:11:50.335075+0000","last_clean":"2024-03-04T16:11:50.335075+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:11:50.335075+0000","last_undegraded":"2024-03-04T16:11:50.335075+0000","last_fullsized":"2024-03-04T16:11:50.335075+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"75'18","reported_seq":437,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333082+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:11:50.333082+0000","last_peered":"2024-03-04T16:11:50.333082+0000","last_clean":"2024-03-04T16:11:50.333082+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:11:50.333082+0000","last_undegraded":"2024-03-04T16:11:50.333082+0000","last_fullsized":"2024-03-04T16:11:50.333082+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267295+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:11:57.267295+0000","last_peered":"2024-03-04T16:11:57.267295+0000","last_clean":"2024-03-04T16:11:57.267295+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:11:57.267295+0000","last_undegraded":"2024-03-04T16:11:57.267295+0000","last_fullsized":"2024-03-04T16:11:57.267295+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"226'12164","reported_seq":15860,"reported_epoch":226,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:11:58.042594+0000","last_change":"2024-03-04T16:11:45.258120+0000","last_active":"2024-03-04T16:11:58.042594+0000","last_peered":"2024-03-04T16:11:58.042594+0000","last_clean":"2024-03-04T16:10:28.209640+0000","last_became_active":"2024-03-04T16:10:30.368524+0000","last_became_peered":"2024-03-04T16:10:30.368524+0000","last_unstale":"2024-03-04T16:11:58.042594+0000","last_undegraded":"2024-03-04T16:11:58.042594+0000","last_fullsized":"2024-03-04T16:11:58.042594+0000","mapping_epoch":195,"log_start":"200'8478","ondisk_log_start":"200'8478","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188481536,"num_objects":2876,"num_object_clones":0,"num_object_copies":8628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1372,"num_objects_unfound":0,"num_objects_dirty":2876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12164,"num_write_kb":481280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1779,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[{"shards":"1,5,6","objects":2876}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":362,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273340+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:11:57.273340+0000","last_peered":"2024-03-04T16:11:57.273340+0000","last_clean":"2024-03-04T16:11:57.273340+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:11:57.273340+0000","last_undegraded":"2024-03-04T16:11:57.273340+0000","last_fullsized":"2024-03-04T16:11:57.273340+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.490215+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":466,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268840+0000","last_change":"2024-03-04T16:08:05.968395+0000","last_active":"2024-03-04T16:11:57.268840+0000","last_peered":"2024-03-04T16:11:57.268840+0000","last_clean":"2024-03-04T16:11:57.268840+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:11:57.268840+0000","last_undegraded":"2024-03-04T16:11:57.268840+0000","last_fullsized":"2024-03-04T16:11:57.268840+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:08:05.968259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:05.968259+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":353,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333041+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:11:50.333041+0000","last_peered":"2024-03-04T16:11:50.333041+0000","last_clean":"2024-03-04T16:11:50.333041+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:11:50.333041+0000","last_undegraded":"2024-03-04T16:11:50.333041+0000","last_fullsized":"2024-03-04T16:11:50.333041+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"75'15","reported_seq":376,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.284233+0000","last_change":"2024-03-04T16:08:09.738935+0000","last_active":"2024-03-04T16:11:57.284233+0000","last_peered":"2024-03-04T16:11:57.284233+0000","last_clean":"2024-03-04T16:11:57.284233+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:11:57.284233+0000","last_undegraded":"2024-03-04T16:11:57.284233+0000","last_fullsized":"2024-03-04T16:11:57.284233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:08:09.738789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.738789+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"226'12358","reported_seq":12487,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.444907+0000","last_change":"2024-03-04T16:08:56.231760+0000","last_active":"2024-03-04T16:11:57.444907+0000","last_peered":"2024-03-04T16:11:57.444907+0000","last_clean":"2024-03-04T16:11:57.444907+0000","last_became_active":"2024-03-04T16:08:56.231431+0000","last_became_peered":"2024-03-04T16:08:56.231431+0000","last_unstale":"2024-03-04T16:11:57.444907+0000","last_undegraded":"2024-03-04T16:11:57.444907+0000","last_fullsized":"2024-03-04T16:11:57.444907+0000","mapping_epoch":167,"log_start":"199'8097","ondisk_log_start":"199'8097","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192675840,"num_objects":2940,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12358,"num_write_kb":489536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":344,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333749+0000","last_change":"2024-03-04T16:10:29.422572+0000","last_active":"2024-03-04T16:11:50.333749+0000","last_peered":"2024-03-04T16:11:50.333749+0000","last_clean":"2024-03-04T16:11:50.333749+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:11:50.333749+0000","last_undegraded":"2024-03-04T16:11:50.333749+0000","last_fullsized":"2024-03-04T16:11:50.333749+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:08:12.467141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:12.467141+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":360,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.273532+0000","last_change":"2024-03-04T16:10:29.344545+0000","last_active":"2024-03-04T16:11:57.273532+0000","last_peered":"2024-03-04T16:11:57.273532+0000","last_clean":"2024-03-04T16:11:57.273532+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:11:57.273532+0000","last_undegraded":"2024-03-04T16:11:57.273532+0000","last_fullsized":"2024-03-04T16:11:57.273532+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.488786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":361,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.267419+0000","last_change":"2024-03-04T16:10:17.472526+0000","last_active":"2024-03-04T16:11:57.267419+0000","last_peered":"2024-03-04T16:11:57.267419+0000","last_clean":"2024-03-04T16:11:57.267419+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:11:57.267419+0000","last_undegraded":"2024-03-04T16:11:57.267419+0000","last_fullsized":"2024-03-04T16:11:57.267419+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"113'16","reported_seq":401,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268514+0000","last_change":"2024-03-04T16:10:29.365556+0000","last_active":"2024-03-04T16:11:57.268514+0000","last_peered":"2024-03-04T16:11:57.268514+0000","last_clean":"2024-03-04T16:11:57.268514+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:11:57.268514+0000","last_undegraded":"2024-03-04T16:11:57.268514+0000","last_fullsized":"2024-03-04T16:11:57.268514+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:10:03.830634+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.830634+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"226'12438","reported_seq":26278,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:58.041752+0000","last_change":"2024-03-04T16:11:16.328482+0000","last_active":"2024-03-04T16:11:58.041752+0000","last_peered":"2024-03-04T16:11:58.041752+0000","last_clean":"2024-03-04T16:11:58.041752+0000","last_became_active":"2024-03-04T16:11:16.308816+0000","last_became_peered":"2024-03-04T16:11:16.308816+0000","last_unstale":"2024-03-04T16:11:58.041752+0000","last_undegraded":"2024-03-04T16:11:58.041752+0000","last_fullsized":"2024-03-04T16:11:58.041752+0000","mapping_epoch":208,"log_start":"200'8722","ondisk_log_start":"200'8722","created":167,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192806912,"num_objects":2942,"num_object_clones":0,"num_object_copies":8826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12438,"num_write_kb":492160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6846,"num_bytes_recovered":448659456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.334809+0000","last_change":"2024-03-04T16:10:29.323919+0000","last_active":"2024-03-04T16:11:50.334809+0000","last_peered":"2024-03-04T16:11:50.334809+0000","last_clean":"2024-03-04T16:11:50.334809+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:11:50.334809+0000","last_undegraded":"2024-03-04T16:11:50.334809+0000","last_fullsized":"2024-03-04T16:11:50.334809+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":355,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.278865+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:11:57.278865+0000","last_peered":"2024-03-04T16:11:57.278865+0000","last_clean":"2024-03-04T16:11:57.278865+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:11:57.278865+0000","last_undegraded":"2024-03-04T16:11:57.278865+0000","last_fullsized":"2024-03-04T16:11:57.278865+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":74,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268080+0000","last_change":"2024-03-04T16:10:29.322011+0000","last_active":"2024-03-04T16:11:57.268080+0000","last_peered":"2024-03-04T16:11:57.268080+0000","last_clean":"2024-03-04T16:11:57.268080+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:11:57.268080+0000","last_undegraded":"2024-03-04T16:11:57.268080+0000","last_fullsized":"2024-03-04T16:11:57.268080+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"75'17","reported_seq":435,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268458+0000","last_change":"2024-03-04T16:10:29.356897+0000","last_active":"2024-03-04T16:11:57.268458+0000","last_peered":"2024-03-04T16:11:57.268458+0000","last_clean":"2024-03-04T16:11:57.268458+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:11:57.268458+0000","last_undegraded":"2024-03-04T16:11:57.268458+0000","last_fullsized":"2024-03-04T16:11:57.268458+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:09:56.769841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.769841+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"225'11680","reported_seq":11807,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:53.390814+0000","last_change":"2024-03-04T16:08:56.232238+0000","last_active":"2024-03-04T16:11:53.390814+0000","last_peered":"2024-03-04T16:11:53.390814+0000","last_clean":"2024-03-04T16:11:53.390814+0000","last_became_active":"2024-03-04T16:08:56.231073+0000","last_became_peered":"2024-03-04T16:08:56.231073+0000","last_unstale":"2024-03-04T16:11:53.390814+0000","last_undegraded":"2024-03-04T16:11:53.390814+0000","last_fullsized":"2024-03-04T16:11:53.390814+0000","mapping_epoch":167,"log_start":"199'7732","ondisk_log_start":"199'7732","created":167,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:55.182208+0000","last_clean_scrub_stamp":"2024-03-04T16:08:55.182208+0000","objects_scrubbed":0,"log_size":3948,"log_dups_size":0,"ondisk_log_size":3948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205258752,"num_objects":3132,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11680,"num_write_kb":473984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":349,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268832+0000","last_change":"2024-03-04T16:10:17.475035+0000","last_active":"2024-03-04T16:11:57.268832+0000","last_peered":"2024-03-04T16:11:57.268832+0000","last_clean":"2024-03-04T16:11:57.268832+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:11:57.268832+0000","last_undegraded":"2024-03-04T16:11:57.268832+0000","last_fullsized":"2024-03-04T16:11:57.268832+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:13.787481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:13.787481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":194,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.270411+0000","last_change":"2024-03-04T16:10:38.935575+0000","last_active":"2024-03-04T16:11:57.270411+0000","last_peered":"2024-03-04T16:11:57.270411+0000","last_clean":"2024-03-04T16:11:57.270411+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:11:57.270411+0000","last_undegraded":"2024-03-04T16:11:57.270411+0000","last_fullsized":"2024-03-04T16:11:57.270411+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:10:38.935466+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:10:38.935466+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:00.582470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":365,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-04T16:11:50.333687+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:11:50.333687+0000","last_peered":"2024-03-04T16:11:50.333687+0000","last_clean":"2024-03-04T16:11:50.333687+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:11:50.333687+0000","last_undegraded":"2024-03-04T16:11:50.333687+0000","last_fullsized":"2024-03-04T16:11:50.333687+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"75'10","reported_seq":384,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-03-04T16:11:57.268209+0000","last_change":"2024-03-04T16:10:29.336876+0000","last_active":"2024-03-04T16:11:57.268209+0000","last_peered":"2024-03-04T16:11:57.268209+0000","last_clean":"2024-03-04T16:11:57.268209+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:11:57.268209+0000","last_undegraded":"2024-03-04T16:11:57.268209+0000","last_fullsized":"2024-03-04T16:11:57.268209+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3124494360,"num_objects":47677,"num_object_clones":0,"num_object_copies":143031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1372,"num_objects_unfound":0,"num_objects_dirty":47677,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195589,"num_write_kb":7784449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66086,"num_bytes_recovered":4330815512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11185963008,"data_stored":11185946720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65304,"ondisk_log_size":65304,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":547,"num_read_kb":542,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":690,"num_read_kb":447,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070579,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1478356,"kb_used_data":1163980,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92266796,"statfs":{"total":95995035648,"available":94481199104,"internally_reserved":0,"allocated":1191915520,"data_stored":1191509415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":321911242},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364919,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574993,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879068,"kb_used_data":1496656,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91866084,"statfs":{"total":95995035648,"available":94070870016,"internally_reserved":0,"allocated":1532575744,"data_stored":1532170811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1670812,"kb_used_data":1342480,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92074340,"statfs":{"total":95995035648,"available":94284124160,"internally_reserved":0,"allocated":1374699520,"data_stored":1374295788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149829,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2681036,"kb_used_data":2198336,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91064116,"statfs":{"total":95995035648,"available":93249654784,"internally_reserved":0,"allocated":2251096064,"data_stored":2250701022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":493943240},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953601,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1659444,"kb_used_data":1343660,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92085708,"statfs":{"total":95995035648,"available":94295764992,"internally_reserved":0,"allocated":1375907840,"data_stored":1375532022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626557,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2053376,"kb_used_data":1693172,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":91691776,"statfs":{"total":95995035648,"available":93892378624,"internally_reserved":0,"allocated":1733808128,"data_stored":1733430757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":368835014},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1340932096,"data_stored":1340932096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1504645120,"data_stored":1504641048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1674579968,"data_stored":1674575896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2227437568,"data_stored":2227437568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1345781760,"data_stored":1345781760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1731399680,"data_stored":1731395608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":191627264,"data_stored":191627264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1169559552,"data_stored":1169555480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:12:00.999 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-04T16:12:01.074 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:01.108 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:01.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:01.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:01.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:01.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.400 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:01.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:12:01.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:12:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:12:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:12:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:12:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:12:01.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:01 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:12:01.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:01.513 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:12:01.513 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:12:01.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:12:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:12:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:12:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:12:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:12:01.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:01 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:12:01.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:01.664 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:01.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:01.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:01.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:01.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:01.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:02.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:02.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:12:02.062 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:02.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:02.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.208 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:02.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:02.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.359 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:02.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:02.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:02.493 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:02.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: 8.e scrub starts 2024-03-04T16:12:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-mon[33619]: pgmap v410: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 1372/140007 objects misplaced (0.980%); 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:02.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:02.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:02.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: 8.e scrub starts 2024-03-04T16:12:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[29779]: pgmap v410: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 1372/140007 objects misplaced (0.980%); 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: 8.e scrub starts 2024-03-04T16:12:02.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:02 smithi038 ceph-mon[38113]: pgmap v410: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 823 op/s; 1372/140007 objects misplaced (0.980%); 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.767 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:02.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:02.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:02.907 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:02.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:03.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.045 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:03.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:03.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:03.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:03.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:12:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.175 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:03.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:03.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.335 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:03.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:03.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:12:03.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 8.e scrub ok 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:12:03.368 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 8.e scrub ok 2024-03-04T16:12:03.369 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:12:03.369 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:03.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:03.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:03.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:03.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:03.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:12:03.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:12:03.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:12:03.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:03.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:03.754 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:03.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:03.821 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 8.e scrub ok 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:12:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:03.860 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:12:03.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:03.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:12:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:03.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:03.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:03.940 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:12:03.987 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:12:04.026 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:12:04.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:04.066 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:12:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.081 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:04.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:04.107 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:12:04.116 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:12:04.117 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:12:04.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:04.216 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:04.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:04.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:04.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.484 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:04.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:04.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:04.617 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:04.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[29779]: pgmap v411: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 768/120147 objects misplaced (0.639%); 6.7 MiB/s, 106 objects/s recovering 2024-03-04T16:12:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:12:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:12:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1406458342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[38113]: pgmap v411: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 768/120147 objects misplaced (0.639%); 6.7 MiB/s, 106 objects/s recovering 2024-03-04T16:12:04.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:12:04.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:12:04.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:04 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1406458342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:04.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:04.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:04 smithi176 ceph-mon[33619]: pgmap v411: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 768/120147 objects misplaced (0.639%); 6.7 MiB/s, 106 objects/s recovering 2024-03-04T16:12:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:04 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:12:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:04 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:12:04.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:04 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1406458342' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:04.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:04.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:04.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:05.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:05.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:05.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.285 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:05.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:05.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:05 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:12:05.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:05 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:12:05.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:05 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:12:05.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:05 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:12:05.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:05.541 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:05.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:05.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:05.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:05.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:05 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:12:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:05 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:12:05.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:05.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:05.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:06.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:12:06.000 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:12:06.000 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-04T16:12:06.000 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:12:06.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:06.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:06.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:06.206 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:06.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:06.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:06.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:06.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:06.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:06.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:06.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[29779]: pgmap v412: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 768/120147 objects misplaced (0.639%); 6.5 MiB/s, 104 objects/s recovering 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:12:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:12:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:12:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:06 smithi038 ceph-mon[38113]: pgmap v412: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 768/120147 objects misplaced (0.639%); 6.5 MiB/s, 104 objects/s recovering 2024-03-04T16:12:06.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:06 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:12:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:06 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:12:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:06 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:12:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:06 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:12:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:06 smithi176 ceph-mon[33619]: pgmap v412: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s; 768/120147 objects misplaced (0.639%); 6.5 MiB/s, 104 objects/s recovering 2024-03-04T16:12:06.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:06.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:06.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:07.002 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:07.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:07.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:07.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.316 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:07.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:07.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:07.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:07.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:07.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:07.698 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.729 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:12:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:07 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:07.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:07 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:12:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:07 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:12:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:07 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:07.864 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:07.968 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:07.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:07.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:08.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:08.129 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:08.231 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.270 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:08.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:08.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.552 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:08.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:08.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[29779]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s; 768/120147 objects misplaced (0.639%); 5.3 MiB/s, 85 objects/s recovering 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:12:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:08 smithi038 ceph-mon[38113]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s; 768/120147 objects misplaced (0.639%); 5.3 MiB/s, 85 objects/s recovering 2024-03-04T16:12:08.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.831 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:08 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:12:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:08 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:12:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:08 smithi176 ceph-mon[33619]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.4 GiB data, 9.5 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s; 768/120147 objects misplaced (0.639%); 5.3 MiB/s, 85 objects/s recovering 2024-03-04T16:12:08.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:08.970 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:08.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:09.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:09.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:09.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:09.238 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:09.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.373 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:09.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:09.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:09.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:09.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.642 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:09.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:12:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:09 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:12:09.744 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:09.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:09 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:12:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:09 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:12:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:09 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:12:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:09 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:12:09.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:09.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:09.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:09.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:10.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:10.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:10.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:10.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:10.303 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:10.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:10.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:10.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:10.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:10.627 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:10.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:10.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:10 smithi038 ceph-mon[29779]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 1.21k op/s; 328/102957 objects misplaced (0.319%); 4.5 MiB/s, 72 objects/s recovering 2024-03-04T16:12:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:10 smithi038 ceph-mon[38113]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 1.21k op/s; 328/102957 objects misplaced (0.319%); 4.5 MiB/s, 72 objects/s recovering 2024-03-04T16:12:10.758 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:10 smithi176 ceph-mon[33619]: pgmap v414: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 917 B/s rd, 0 B/s wr, 1.21k op/s; 328/102957 objects misplaced (0.319%); 4.5 MiB/s, 72 objects/s recovering 2024-03-04T16:12:10.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:10.901 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:10.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:11.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:11.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:11.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:11.169 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:11.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:11.312 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:11.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:11.427 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.459 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:11.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:11.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:11.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:11.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:11.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:11 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:12:11.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:11 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:12:11.703 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:11 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:12:11.703 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:11 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:12:11.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:11.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:11.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:11 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:12:11.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:11 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:12:11.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:11.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:11.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:12.118 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:12.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:12.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:12.277 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:12.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:12.420 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:12:12.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.563 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:12.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:12.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:12.692 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[29779]: pgmap v415: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 328/102957 objects misplaced (0.319%); 4.2 MiB/s, 67 objects/s recovering 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3770651685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[29779]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:12:12.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:12:12.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[38113]: pgmap v415: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 328/102957 objects misplaced (0.319%); 4.2 MiB/s, 67 objects/s recovering 2024-03-04T16:12:12.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3770651685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:12 smithi038 ceph-mon[38113]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:12.809 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:12:12.809 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:12:12.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-mon[33619]: pgmap v415: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.1 GiB data, 8.4 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 328/102957 objects misplaced (0.319%); 4.2 MiB/s, 67 objects/s recovering 2024-03-04T16:12:12.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3770651685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-mon[33619]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:12.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:12.920 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:12.944 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:12.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:13.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.071 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:13.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:13.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:13.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:13.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:12:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:13.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:13.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:13.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:13.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:13.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:13.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:13.465 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:13.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:13.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:13.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: osdmap e228: 8 total, 8 up, 6 in 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:12:13.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:12:13.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:13.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:13.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: osdmap e228: 8 total, 8 up, 6 in 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:13.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:13.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:13.778 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:13.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: osdmap e228: 8 total, 8 up, 6 in 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:12:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:13.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:13.860 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:13.860 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:13.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:13.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:14.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:14.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:14.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:14.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:14.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:14.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:14.438 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:14.540 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.573 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:14.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:14.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.712 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:14.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:14.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:12:14.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:12:14.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: osdmap e229: 8 total, 8 up, 6 in 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: pgmap v418: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 5.2 MiB/s, 82 objects/s recovering 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 6.13 deep-scrub starts 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[29779]: 6.13 deep-scrub ok 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: osdmap e229: 8 total, 8 up, 6 in 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: pgmap v418: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 5.2 MiB/s, 82 objects/s recovering 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:12:14.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 6.13 deep-scrub starts 2024-03-04T16:12:14.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:14 smithi038 ceph-mon[38113]: 6.13 deep-scrub ok 2024-03-04T16:12:14.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:14.850 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: osdmap e229: 8 total, 8 up, 6 in 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: pgmap v418: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s; 5.2 MiB/s, 82 objects/s recovering 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:12:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:12:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 6.13 deep-scrub starts 2024-03-04T16:12:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:14 smithi176 ceph-mon[33619]: 6.13 deep-scrub ok 2024-03-04T16:12:14.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:14.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:14.987 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:14.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:15.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:15.122 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:15.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.255 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:15.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:15.395 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:15.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:15.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:15.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:15.661 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:15.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: osdmap e230: 8 total, 8 up, 6 in 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:12:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: osdmap e230: 8 total, 8 up, 6 in 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:12:15.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:15 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:12:15.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:15.794 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: osdmap e230: 8 total, 8 up, 6 in 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:12:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:12:15.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:15 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:12:15.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:15.927 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:15.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:16.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:16.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:16.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:16.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:163741 7.9 7:91e9475d:::benchmark_data_smithi038_77863_object45473:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:05.199802+0000", 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.09222584600002, 2024-03-04T16:12:16.298 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1279827139999998, 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 163741 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199802+0000", 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199802+0000", 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199804+0000", 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2409999999999998e-06 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.300 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199812+0000", 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.199e-06 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199823+0000", 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1646e-05 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.199833+0000", 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0294e-05 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.301 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.200939+0000", 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011051520000000001 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.201003+0000", 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4657000000000001e-05 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.302 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206556+0000", 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0055529590000001 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206570+0000", 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3861e-05 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.325952+0000", 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1193814609999999 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.303 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.327785+0000", 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018332439999999999 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:163756 7.9 7:9c8111ed:::benchmark_data_smithi038_77863_object45488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:12:16.304 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:05.216589+0000", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.07543941, 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1352352240000001, 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 163756 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216589+0000", 2024-03-04T16:12:16.305 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216589+0000", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216590+0000", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.88e-06 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216596+0000", 2024-03-04T16:12:16.306 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8080000000000001e-06 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216610+0000", 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3549e-05 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216618+0000", 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2709999999999992e-06 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.307 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216745+0000", 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012679999999999999 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.216757+0000", 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1695999999999999e-05 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206599+0000", 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98984221400000005 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.308 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206608+0000", 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1949999999999992e-06 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.328197+0000", 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1215891550000001 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.309 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.329205+0000", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010083589999999999 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,4,7", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.329571+0000", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036581500000000002 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351105+0000", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021533652 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.310 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351159+0000", 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3667e-05 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351210+0000", 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1771999999999998e-05 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351624+0000", 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000413461 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.311 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351695+0000", 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0926999999999994e-05 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351824+0000", 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012900300000000001 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.312 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:163762 7.9 7:9e0c9125:::benchmark_data_smithi038_77863_object45494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:05.230112+0000", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.06191575999998, 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.12186253, 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 163762 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.313 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230112+0000", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230112+0000", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230115+0000", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7709999999999998e-06 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.314 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230123+0000", 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0409999999999998e-06 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230141+0000", 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7966999999999999e-05 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230151+0000", 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0294e-05 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.315 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230261+0000", 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110016 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.230273+0000", 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1936999999999999e-05 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.316 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206634+0000", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97636088099999996 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206648+0000", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.367e-05 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.341316+0000", 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1346682960000001 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.317 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.341460+0000", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014395099999999999 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,4,7", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.341603+0000", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014319699999999999 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351281+0000", 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0096782699999999992 2024-03-04T16:12:16.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351350+0000", 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8635999999999998e-05 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351402+0000", 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1981000000000003e-05 2024-03-04T16:12:16.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351830+0000", 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00042780200000000002 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351868+0000", 2024-03-04T16:12:16.320 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7744000000000003e-05 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351975+0000", 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107076 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.321 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:163766 7.9 7:9d9f83ed:::benchmark_data_smithi038_77863_object45498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:05.234306+0000", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.057722136, 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.129556145, 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 163766 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.322 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234306+0000", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234306+0000", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234308+0000", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.232e-06 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234315+0000", 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1829999999999997e-06 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234329+0000", 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.347e-05 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234336+0000", 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0960000000000003e-06 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.324 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234418+0000", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2682000000000006e-05 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.234432+0000", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3147e-05 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206673+0000", 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97224095099999996 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.325 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206680+0000", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8429999999999993e-06 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.345070+0000", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1383891349999999 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.345225+0000", 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000155606 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.326 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,4,7", 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.345372+0000", 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000147155 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.327 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:16.328 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351455+0000", 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060825619999999997 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351507+0000", 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.202e-05 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351983+0000", 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00047647599999999998 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.329 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.363145+0000", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011161752 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.363733+0000", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00058828099999999996 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.363862+0000", 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012855399999999999 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.330 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:163769 7.9 7:9229bfaf:::benchmark_data_smithi038_77863_object45501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:05.244156+0000", 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 311.047871692, 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1273286179999999, 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.331 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:12:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:12:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 163769 2024-03-04T16:12:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.332 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244156+0000", 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.333 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244156+0000", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244159+0000", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.367e-06 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244166+0000", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5220000000000001e-06 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.334 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244181+0000", 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5316e-05 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244192+0000", 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0069e-05 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244251+0000", 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9938000000000001e-05 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.335 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:05.244261+0000", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.183e-06 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206704+0000", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96244386599999998 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:06.206713+0000", 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3459999999999994e-06 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.336 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.347756+0000", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.141042761 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.347891+0000", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013574400000000001 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,4,7", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.348043+0000", 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015146499999999999 2024-03-04T16:12:16.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351558+0000", 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003514905 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351607+0000", 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8965999999999998e-05 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.338 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.351987+0000", 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00038059500000000001 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.371065+0000", 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019077502999999999 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.339 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.371352+0000", 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00028716499999999998 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:07.371485+0000", 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013290700000000001 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114765 8.3 8:c944b839:::benchmark_data_smithi038_107057_object114764:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:12:16.340 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.214479+0000", 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 108.077548589, 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1586476509999999, 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114765 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.341 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214479+0000", 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214479+0000", 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214481+0000", 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0810000000000001e-06 2024-03-04T16:12:16.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214507+0000", 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5565e-05 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214519+0000", 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2092e-05 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.343 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214527+0000", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4139999999999994e-06 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214586+0000", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8152999999999997e-05 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214718+0000", 2024-03-04T16:12:16.344 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013222 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5,6", 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.214859+0000", 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000141528 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.219391+0000", 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045311850000000001 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.345 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.221982+0000", 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025913749999999999 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.244760+0000", 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.022777577 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.244775+0000", 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5130000000000001e-05 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.346 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.238820+0000", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99404583400000002 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.238833+0000", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2835e-05 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.373042+0000", 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1342085829999999 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.347 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.373127+0000", 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5079000000000003e-05 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114770 8.7 8:edab5fb8:::benchmark_data_smithi038_107057_object114769:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.223341+0000", 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 108.068686924, 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1083284720000002, 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114770 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223341+0000", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.349 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223341+0000", 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223343+0000", 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223366+0000", 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2952e-05 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.350 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223377+0000", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1745e-05 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223385+0000", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0339999999999997e-06 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223469+0000", 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3168000000000001e-05 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.351 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.223603+0000", 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013398800000000001 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.224557+0000", 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00095485100000000005 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.352 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.250751+0000", 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.026193873999999999 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.250765+0000", 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.365e-05 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.353 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237849+0000", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98708404500000002 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237863+0000", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.433e-05 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.311043+0000", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0731791850000001 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-04T16:12:16.354 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.311848+0000", 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080517000000000002 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331582+0000", 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019734087000000001 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331669+0000", 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7632000000000004e-05 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114773 8.7 8:e537915c:::benchmark_data_smithi038_107057_object114772:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.232419+0000", 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 108.059608983, 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.099817732, 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114773 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232419+0000", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232419+0000", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232421+0000", 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6700000000000001e-06 2024-03-04T16:12:16.357 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232449+0000", 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8354e-05 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232459+0000", 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0526000000000001e-05 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.359 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.232470+0000", 2024-03-04T16:12:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0712e-05 2024-03-04T16:12:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.251805+0000", 2024-03-04T16:12:16.363 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.019335031999999999 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.251815+0000", 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0132999999999999e-05 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.364 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237894+0000", 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98607868600000004 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237902+0000", 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8450000000000003e-06 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.312671+0000", 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0747694729999999 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.312835+0000", 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016354299999999999 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4,6", 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.313364+0000", 2024-03-04T16:12:16.366 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052859900000000004 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.330856+0000", 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.017492133 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.367 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331749+0000", 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089301400000000003 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331787+0000", 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8661000000000001e-05 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331937+0000", 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014941499999999999 2024-03-04T16:12:16.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332112+0000", 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017506100000000001 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332237+0000", 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012487500000000001 2024-03-04T16:12:16.369 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114787 8.7 8:e47f5ab0:::benchmark_data_smithi038_107057_object114786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.263059+0000", 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 108.028969132, 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0699311269999998, 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.370 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114787 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263059+0000", 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263059+0000", 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.371 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263061+0000", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4650000000000001e-06 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263106+0000", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5065000000000003e-05 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263119+0000", 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2604e-05 2024-03-04T16:12:16.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263136+0000", 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6826000000000001e-05 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263336+0000", 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000199843 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.373 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.263351+0000", 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5702e-05 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237929+0000", 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97457789900000003 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.374 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.237937+0000", 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2989999999999997e-06 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.323521+0000", 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.085583585 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.375 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.323670+0000", 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014916300000000001 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4,6", 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.324104+0000", 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000433953 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331709+0000", 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076049580000000002 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.376 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332279+0000", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056948199999999995 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332672+0000", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039358999999999998 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332696+0000", 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3489000000000002e-05 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332865+0000", 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016893 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332990+0000", 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012527400000000001 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.378 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114791 8.3 8:cb8b7900:::benchmark_data_smithi038_107057_object114790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.267454+0000", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 108.02457438, 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.124068426, 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114791 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.379 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267454+0000", 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267454+0000", 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267456+0000", 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1830000000000001e-06 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.380 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267485+0000", 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9666999999999999e-05 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267499+0000", 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3798000000000001e-05 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.381 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267509+0000", 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8640000000000008e-06 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267586+0000", 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6877000000000003e-05 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.382 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.267598+0000", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1656e-05 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.238867+0000", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97126963200000005 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.238877+0000", 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.995e-06 2024-03-04T16:12:16.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.373813+0000", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1349362160000001 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.373985+0000", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171309 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.374436+0000", 2024-03-04T16:12:16.384 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045127299999999997 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391150+0000", 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.016713849999999999 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391246+0000", 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6558000000000006e-05 2024-03-04T16:12:16.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391296+0000", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9218000000000002e-05 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391334+0000", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8773999999999998e-05 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391399+0000", 2024-03-04T16:12:16.386 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4494999999999998e-05 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391522+0000", 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012306099999999999 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114794 8.3 8:c42f2b50:::benchmark_data_smithi038_107057_object114793:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.433212+0000", 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 106.858815824, 2024-03-04T16:12:16.387 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95959000400000005, 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114794 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433212+0000", 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433212+0000", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433214+0000", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.84e-06 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433240+0000", 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5769e-05 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433252+0000", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2303000000000001e-05 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433258+0000", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3130000000000004e-06 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433322+0000", 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4103999999999995e-05 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.433336+0000", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.33e-05 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.385425+0000", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95208919299999994 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.385568+0000", 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000143566 2024-03-04T16:12:16.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.385744+0000", 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000175859 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391260+0000", 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055156179999999999 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.392 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391565+0000", 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030514099999999998 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.391619+0000", 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4358999999999999e-05 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.392623+0000", 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010038479999999999 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.393 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.392684+0000", 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0534e-05 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.392802+0000", 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118257 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114801 8.3 8:c3872e77:::benchmark_data_smithi038_107057_object114800:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:12:16.394 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.453199+0000", 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 106.83882855100001, 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94232977900000003, 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114801 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.395 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453199+0000", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453199+0000", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453201+0000", 2024-03-04T16:12:16.396 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9410000000000001e-06 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453237+0000", 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5259000000000002e-05 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453252+0000", 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5082000000000001e-05 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.397 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453260+0000", 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2449999999999998e-06 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453333+0000", 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2706000000000005e-05 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.453348+0000", 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5033e-05 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.398 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.388012+0000", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93466476600000004 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.388830+0000", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00081735300000000005 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.389001+0000", 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017085099999999999 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.392846+0000", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038458490000000001 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.393570+0000", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072393700000000002 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.393595+0000", 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4994000000000002e-05 2024-03-04T16:12:16.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.395344+0000", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001748388 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.395414+0000", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9820999999999994e-05 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.395529+0000", 2024-03-04T16:12:16.401 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115554 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215065 8.3 8:c7be943b:::benchmark_data_smithi038_107057_object93430:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.528782+0000", 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7632458710000001, 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95752190100000001, 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.402 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215065 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528782+0000", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528782+0000", 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528784+0000", 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.815e-06 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528790+0000", 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4659999999999996e-06 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.404 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528802+0000", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1595e-05 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528811+0000", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6309999999999992e-06 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528965+0000", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015416 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.405 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.528975+0000", 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0658e-05 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.481796+0000", 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95282064200000005 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.481889+0000", 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3348000000000001e-05 2024-03-04T16:12:16.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.481996+0000", 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106727 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.408 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486104+0000", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041078 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486152+0000", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8019000000000002e-05 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486245+0000", 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3182000000000003e-05 2024-03-04T16:12:16.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486279+0000", 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4037e-05 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486304+0000", 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4820999999999998e-05 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.410 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215080 8.3 8:cb489614:::benchmark_data_smithi038_107057_object93445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.539805+0000", 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7522229820000002, 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95064068199999996, 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215080 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.411 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539805+0000", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539805+0000", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539807+0000", 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9870000000000002e-06 2024-03-04T16:12:16.412 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539813+0000", 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3910000000000003e-06 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539829+0000", 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6120000000000002e-05 2024-03-04T16:12:16.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539838+0000", 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6079999999999996e-06 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539895+0000", 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6708e-05 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.414 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.539903+0000", 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2940000000000006e-06 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.483856+0000", 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.943953131 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.483928+0000", 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1528999999999999e-05 2024-03-04T16:12:16.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.484024+0000", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6663999999999995e-05 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486141+0000", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021171219999999999 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486208+0000", 2024-03-04T16:12:16.416 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6310999999999996e-05 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490343+0000", 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041352519999999999 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490398+0000", 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5331000000000003e-05 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.417 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490446+0000", 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7234000000000002e-05 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215110 8.3 8:ce25a2b9:::benchmark_data_smithi038_107057_object93475:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.565566+0000", 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7264620800000001, 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92510630400000005, 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.418 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215110 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565566+0000", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565566+0000", 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.419 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565568+0000", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.761e-06 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565573+0000", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7810000000000002e-06 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565589+0000", 2024-03-04T16:12:16.420 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5724999999999999e-05 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.565597+0000", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0330000000000008e-06 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.566392+0000", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00079498700000000004 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.566401+0000", 2024-03-04T16:12:16.421 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0860000000000008e-06 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.485143+0000", 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.91874173599999998 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.485209+0000", 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6141000000000003e-05 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.422 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.485277+0000", 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7787999999999995e-05 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490296+0000", 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0050194910000000001 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490521+0000", 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000224343 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.423 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490578+0000", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7673000000000002e-05 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490634+0000", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5924999999999997e-05 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490672+0000", 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7833999999999997e-05 2024-03-04T16:12:16.424 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215125 8.3 8:cc19e1cb:::benchmark_data_smithi038_107057_object93490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.575279+0000", 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7167489319999998, 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.91561880799999995, 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.425 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215125 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575279+0000", 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575279+0000", 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.426 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575281+0000", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6050000000000001e-06 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575286+0000", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0069999999999997e-06 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575295+0000", 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6630000000000005e-06 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.427 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575303+0000", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5900000000000002e-06 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575354+0000", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0683999999999998e-05 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.575362+0000", 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1289999999999996e-06 2024-03-04T16:12:16.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486355+0000", 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.91099303300000001 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486461+0000", 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105925 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.429 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.486528+0000", 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7271e-05 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490302+0000", 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037737539999999998 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490737+0000", 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00043501100000000001 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.430 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490804+0000", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7210000000000002e-05 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490856+0000", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2506999999999997e-05 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490898+0000", 2024-03-04T16:12:16.431 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1418999999999998e-05 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215138 8.3 8:cb43f7b5:::benchmark_data_smithi038_107057_object93503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.584244+0000", 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7077843860000002, 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.91096112299999998, 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.432 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215138 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584244+0000", 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584244+0000", 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.433 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584245+0000", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.319e-06 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584249+0000", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5050000000000001e-06 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584259+0000", 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7170000000000003e-06 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.434 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584267+0000", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5340000000000002e-06 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584321+0000", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4836999999999997e-05 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.584329+0000", 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0160000000000008e-06 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.435 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.488337+0000", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.90400748399999997 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.488451+0000", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114407 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.488570+0000", 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000118792 2024-03-04T16:12:16.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490685+0000", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0021150890000000001 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.490989+0000", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030361500000000003 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495124+0000", 2024-03-04T16:12:16.437 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004134665 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495165+0000", 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1418000000000002e-05 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495205+0000", 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9725000000000003e-05 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.438 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215142 8.3 8:ce9844d4:::benchmark_data_smithi038_107057_object93507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.587255+0000", 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.7047733049999998, 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.91719947899999998, 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215142 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.439 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587255+0000", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587255+0000", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587256+0000", 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.731e-06 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.440 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587262+0000", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8490000000000002e-06 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587273+0000", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1080999999999999e-05 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587281+0000", 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9470000000000003e-06 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.441 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587337+0000", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5998999999999997e-05 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.587348+0000", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0797e-05 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.491110+0000", 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.90376166599999996 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.442 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.491252+0000", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014252500000000001 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.491389+0000", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013662200000000001 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495223+0000", 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003834465 2024-03-04T16:12:16.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.498567+0000", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033441970000000001 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.504349+0000", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057814930000000004 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.504405+0000", 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6075000000000001e-05 2024-03-04T16:12:16.444 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.504454+0000", 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9032000000000001e-05 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215194 8.3 8:cb355413:::benchmark_data_smithi038_107057_object93559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.625225+0000", 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.666802933, 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88721460600000002, 2024-03-04T16:12:16.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215194 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625225+0000", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.446 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625225+0000", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625227+0000", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.606e-06 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625233+0000", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0090000000000004e-06 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625244+0000", 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1882000000000001e-05 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625254+0000", 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1020000000000003e-06 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625312+0000", 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8155999999999997e-05 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.448 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.625321+0000", 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9700000000000005e-06 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.449 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:16.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495255+0000", 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.869934184 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.450 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495356+0000", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101007 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.495485+0000", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012926300000000001 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.498663+0000", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003178228 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.451 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.498670+0000", 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9890000000000003e-06 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512353+0000", 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013682537 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512398+0000", 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5157000000000002e-05 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.452 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512440+0000", 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1516e-05 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:215202 8.3 8:cde0ac0e:::benchmark_data_smithi038_107057_object93567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:12.631188+0000", 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 3.6608402249999998, 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88158986500000003, 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:12:16.453 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 215202 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631188+0000", 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:12:16.454 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631188+0000", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631189+0000", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6649999999999999e-06 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631194+0000", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9690000000000002e-06 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:12:16.455 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631207+0000", 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2860000000000001e-05 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631215+0000", 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5120000000000002e-06 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631267+0000", 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2389000000000003e-05 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.456 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:12.631275+0000", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2500000000000006e-06 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.496830+0000", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86555477700000005 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.496923+0000", 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3145000000000004e-05 2024-03-04T16:12:16.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.496997+0000", 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3719000000000003e-05 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.498700+0000", 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001702949 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:12:16.458 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.500514+0000", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001813624 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512585+0000", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012071828 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512730+0000", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000144208 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:12:16.459 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:13.512778+0000", 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7970000000000003e-05 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:16.460 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:12:16.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:16.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:16.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.727 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:16.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[29779]: pgmap v420: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[29779]: 4.1b deep-scrub starts 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[29779]: 4.1b deep-scrub ok 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[38113]: pgmap v420: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[38113]: 4.1b deep-scrub starts 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[38113]: 4.1b deep-scrub ok 2024-03-04T16:12:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:12:16.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:16 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:12:16.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:16 smithi176 ceph-mon[33619]: pgmap v420: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-04T16:12:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:16 smithi176 ceph-mon[33619]: 4.1b deep-scrub starts 2024-03-04T16:12:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:16 smithi176 ceph-mon[33619]: 4.1b deep-scrub ok 2024-03-04T16:12:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:16 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:12:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:16 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:12:16.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:16.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:16.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:17.026 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:17.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.173 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:17.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:17.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:17.313 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:17.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.450 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:17.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:17.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:17.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:17.683 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:17.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[29779]: osdmap e231: 8 total, 8 up, 6 in 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:17 smithi038 ceph-mon[38113]: osdmap e231: 8 total, 8 up, 6 in 2024-03-04T16:12:17.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:17.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:17 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:17 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:17 smithi176 ceph-mon[33619]: osdmap e231: 8 total, 8 up, 6 in 2024-03-04T16:12:17.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:17.974 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:17.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:18.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:12:18.076 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.107 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:18.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:18.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.270 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:18.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:18.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:18.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:18.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:18.573 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:12:18.615 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:12:18.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:18.653 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:12:18.665 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:18.696 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:12:18.737 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:12:18.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[29779]: 8.6 scrub starts 2024-03-04T16:12:18.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[29779]: pgmap v422: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1018 B/s rd, 0 B/s wr, 1.21k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-03-04T16:12:18.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:12:18.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[38113]: 8.6 scrub starts 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[38113]: pgmap v422: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1018 B/s rd, 0 B/s wr, 1.21k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:12:18.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:18 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:12:18.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:18.783 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:12:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.804 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:18.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:18.824 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:12:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:18 smithi176 ceph-mon[33619]: 8.6 scrub starts 2024-03-04T16:12:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:18 smithi176 ceph-mon[33619]: pgmap v422: 148 pgs: 1 peering, 147 active+clean; 1.7 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1018 B/s rd, 0 B/s wr, 1.21k op/s; 3.8 MiB/s, 60 objects/s recovering 2024-03-04T16:12:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:18 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:12:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:18 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:12:18.863 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:12:18.866 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:12:18.906 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:18.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:18.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:18.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:19.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:19.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:19.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:19.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:19.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:19.446 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:19.479 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:19.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:19.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:19.618 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:19.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 8.6 scrub ok 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1296543067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: 8.d deep-scrub starts 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:19.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 8.6 scrub ok 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1296543067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: 8.d deep-scrub starts 2024-03-04T16:12:19.711 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:19.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:19.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:19.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:19 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:19.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:19.722 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:19.749 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:19.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:19.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 8.6 scrub ok 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1296543067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: 8.d deep-scrub starts 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:19.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:19 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:19.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:19.889 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:19.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:19.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:20.024 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:20.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:20.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:20.161 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:20.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.289 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:20.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:20.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.426 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:20.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:20.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:20.557 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:20.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:20.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:20.795 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.829 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:20.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:12:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:12:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: pgmap v423: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: osdmap e232: 8 total, 8 up, 6 in 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:12:20.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:20 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:12:20.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:20.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:20.965 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:20.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: pgmap v423: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: osdmap e232: 8 total, 8 up, 6 in 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:12:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: pgmap v423: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 849 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: osdmap e232: 8 total, 8 up, 6 in 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:12:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:20 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:12:21.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:21.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:21.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.240 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:21.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:21.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.372 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:21.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:21.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:21.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:21.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:21.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:21.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:12:21.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:21.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:21 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:12:21.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:21 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:12:21.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:21 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:12:21.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:21 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:12:21.853 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:21.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:21 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:12:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:21 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:12:21.903 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:12:21.903 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:12:21.903 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 232, 2024-03-04T16:12:21.903 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:12:21.903 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:12:19.514559+0000", 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:10:28.206965+0000", 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5799936, 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:12:21.904 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:12:21.905 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 8, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.906 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:12:21.907 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "185", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.908 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.909 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.910 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:12:21.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.913 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.914 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:12:21.915 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:12:21.916 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.916 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.916 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.916 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.917 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.918 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.919 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.920 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:12:21.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6500000953674316, 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6500000953674316, 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-04T16:12:21.922 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.925 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:21.926 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.927 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.928 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.930 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.931 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.932 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.933 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.934 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.75, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.75, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.309999942779541, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.309999942779541, 2024-03-04T16:12:21.935 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:12:21.936 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.937 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:21.938 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.939 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:12:21.940 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.941 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.942 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.943 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.944 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5099999904632568, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5099999904632568, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:21.945 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.946 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.947 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.948 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.949 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.950 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:21.951 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:12:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:12:21.952 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.954 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.955 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.956 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.957 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.958 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 8, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_1", 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:08:54.571316+0000", 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:21.960 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:21.961 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "171", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:21.962 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:21.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:21.964 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:12:21.965 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:21.966 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 224, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.967 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.969 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.970 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 228, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:21.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.972 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:21.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:12:21.974 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 140, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.975 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:21.977 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.978 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.979 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 205, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.980 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.981 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:21.982 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.983 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 209, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.984 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.985 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.986 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:21.987 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.988 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:12:21.989 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:21.991 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 222, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 190, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:21.996 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:12:21.997 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:21.998 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:21.999 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 218, 2024-03-04T16:12:22.000 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:22.001 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.002 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:22.003 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:16.214756+0000", 2024-03-04T16:12:22.004 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:18.787310+0000", 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:12:22.005 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:57.712298+0000", 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.006 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:27.766016+0000", 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:12.599257+0000", 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.007 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:14.089999+0000", 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.008 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:38.936878+0000", 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:18.722753+0000", 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:22.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.7", 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 1, 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 7 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 5, 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 0 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.010 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:22.011 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:12:22.012 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:12:22.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:22.014 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:12:22.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:22.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:22.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:22.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.192 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:22.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:22.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:22.326 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:22.428 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:22.453 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:22.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:22.589 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:22.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:22.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:22.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:22.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: pgmap v425: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 920 op/s 2024-03-04T16:12:22.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3252007247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:22.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:22.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:22.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:12:22.845 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: pgmap v425: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 920 op/s 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3252007247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:22.847 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:22.847 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:22.847 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:12:22.847 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:22 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:12:22.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: pgmap v425: 148 pgs: 148 active+clean; 1.3 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 731 B/s rd, 0 B/s wr, 920 op/s 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3252007247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:12:22.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:22 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:12:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:22.860 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:22.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:22.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.001 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:23.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:23.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:23.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:23.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:23.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:23.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:23.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.410 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:23.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:23.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.541 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:23.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:23.568 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:23.571 INFO:tasks.thrashosds.thrasher:j is {'epoch': 232, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:12:19.514559+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:10:28.206965+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-04T16:08:54.571316+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '171', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 228, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 205, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:16.214756+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:18.787310+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:11:57.712298+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:11:27.766016+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:12.599257+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:14.089999+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:11:38.936878+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:18.722753+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 7}, {'from': 5, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:12:23.571 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.7 2024-03-04T16:12:23.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd rm-pg-upmap-items 4.7 2024-03-04T16:12:23.634 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:23.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.718 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:23.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:23.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]': finished 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: osdmap e233: 8 total, 8 up, 6 in 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:23.848 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]': finished 2024-03-04T16:12:23.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-03-04T16:12:23.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: osdmap e233: 8 total, 8 up, 6 in 2024-03-04T16:12:23.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:12:23.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:12:23.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.13", "id": [0, 5]}]': finished 2024-03-04T16:12:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-03-04T16:12:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: osdmap e233: 8 total, 8 up, 6 in 2024-03-04T16:12:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:12:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:12:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:23.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:23.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:23.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:23.964 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:24.005 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:24.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:24.141 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.178 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:24.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:24.225 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:24.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:24.314 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:24.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:24.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:24.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:24.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:24.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.722 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:24.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: pgmap v427: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.80k op/s 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: osdmap e234: 8 total, 8 up, 6 in 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:12:24.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: pgmap v427: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.80k op/s 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: osdmap e234: 8 total, 8 up, 6 in 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:12:24.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:24 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:12:24.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:12:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:12:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: pgmap v427: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.80k op/s 2024-03-04T16:12:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: osdmap e234: 8 total, 8 up, 6 in 2024-03-04T16:12:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:12:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:12:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:12:24.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:24 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:12:24.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:24.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:24.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:24.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:25.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:25.129 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:25.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:25.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:25.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:25.395 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:25.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:25.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:25.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:25.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:25.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:25.805 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 8.d deep-scrub ok 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:12:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:12:25.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:25 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:12:25.907 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:25.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:25.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 8.d deep-scrub ok 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:12:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 8.d deep-scrub ok 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:12:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:25 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:12:26.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:26.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:26.181 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:26.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:26.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:26.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:26.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:26.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:26.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:26.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:26.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:26.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:26.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:26.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:12:26.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:12:26.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:12:26.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:12:26.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:12:26.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:26 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:12:26.902 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:26.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:26.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:27.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.043 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:27.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: pgmap v429: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:12:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:12:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:12:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:12:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:12:27.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:12:27.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:26 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:12:27.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.176 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:27.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:27.277 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:27.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:27.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:27.448 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:27.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:27.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:27.683 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.716 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:27.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 8.9 scrub starts 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 8.9 scrub ok 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 8.5 scrub starts 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:12:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[29779]: 8.5 scrub ok 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 8.9 scrub starts 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 8.9 scrub ok 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 8.5 scrub starts 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:12:27.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:27 smithi038 ceph-mon[38113]: 8.5 scrub ok 2024-03-04T16:12:27.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:27.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:27.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:27.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:28.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 8.9 scrub starts 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 8.9 scrub ok 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 8.5 scrub starts 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:12:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:27 smithi176 ceph-mon[33619]: 8.5 scrub ok 2024-03-04T16:12:28.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:28.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:28.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:28.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:28.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:28.397 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:28.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:28.537 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:28.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:28.652 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:12:28.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:28.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:28.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:28.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:28.947 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:28.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[29779]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 643 B/s rd, 0 B/s wr, 771 op/s 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1770153918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[29779]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[38113]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 643 B/s rd, 0 B/s wr, 771 op/s 2024-03-04T16:12:28.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1770153918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:28.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:28 smithi038 ceph-mon[38113]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:29.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:28 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:12:29.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:28 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:12:29.010 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:28 smithi176 ceph-mon[33619]: pgmap v430: 148 pgs: 1 active+clean+scrubbing+deep, 2 peering, 145 active+clean; 966 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 643 B/s rd, 0 B/s wr, 771 op/s 2024-03-04T16:12:29.010 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:28 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1770153918' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:29.010 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:28 smithi176 ceph-mon[33619]: from='client.35234 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:12:29.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:29.089 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:29.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:29.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:29.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:29.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:29.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:29.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:29.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:29.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:29.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:29.636 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:29.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.774 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:29.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:29.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:29.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:29.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[29779]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[29779]: osdmap e235: 8 total, 8 up, 6 in 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[38113]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:12:29.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:29 smithi038 ceph-mon[38113]: osdmap e235: 8 total, 8 up, 6 in 2024-03-04T16:12:30.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:30.043 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:30.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:29 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:30.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:29 smithi176 ceph-mon[33619]: from='client.35234 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:12:30.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:29 smithi176 ceph-mon[33619]: osdmap e235: 8 total, 8 up, 6 in 2024-03-04T16:12:30.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:30.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:30.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:30.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:30.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.466 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:30.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:30.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:30.608 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:30.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:30.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:30.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:30 smithi038 ceph-mon[29779]: pgmap v432: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.79k op/s 2024-03-04T16:12:30.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:30 smithi038 ceph-mon[38113]: pgmap v432: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.79k op/s 2024-03-04T16:12:30.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:30.879 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:30.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:30.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:30.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:31.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:31.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:30 smithi176 ceph-mon[33619]: pgmap v432: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 738 B/s rd, 0 B/s wr, 1.79k op/s 2024-03-04T16:12:31.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:31.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:31.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:31.408 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:31.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:31.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:31.691 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:31.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:31.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:31.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:31.973 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:31.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:32.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:32.113 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:32.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:32.203 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:12:32.203 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:12:32.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.244 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:32.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:32.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:32.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:32.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:32.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:32.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:12:32.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:12:32.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:12:32.576 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:12:32.617 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:12:32.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:32.650 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:32.655 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:12:32.697 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:12:32.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:32 smithi038 ceph-mon[29779]: pgmap v433: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:12:32.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:32 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/89459071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:32.743 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:12:32.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:32 smithi038 ceph-mon[38113]: pgmap v433: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:12:32.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:32 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/89459071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:32.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:12:32.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:12:32.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:12:32.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:12:32.751 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:12:32.752 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:12:32.752 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:12:32.752 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:12:32.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:32.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:32.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:32.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:32.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:32.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:33.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:33.077 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:32 smithi176 ceph-mon[33619]: pgmap v433: 148 pgs: 148 active+clean; 528 MiB data, 3.9 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:12:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:32 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/89459071' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:33.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:33.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:12:33.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:33.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:33.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:33.408 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:33.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:33.509 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:33.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:33.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:33.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:12:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:33.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:33.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:33.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:33.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:33.865 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:33.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:33.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:33.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:34.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:34.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:34.120 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Removed 121632 objects 2024-03-04T16:12:34.120 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Clean up completed and total clean up time :115.025 2024-03-04T16:12:34.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.246 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:34.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:34.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:34.476 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:34.477 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:34.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:34.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:34.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:34.697 INFO:teuthology.orchestra.run.smithi038.stderr:clear 4.7 pg_upmap_items mapping 2024-03-04T16:12:34.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:34.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.756 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:34.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:34.883 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:34.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:34.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:34.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:34 smithi038 ceph-mon[29779]: pgmap v434: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-04T16:12:34.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]: dispatch 2024-03-04T16:12:34.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:34 smithi038 ceph-mon[38113]: pgmap v434: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-04T16:12:34.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]: dispatch 2024-03-04T16:12:35.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:35.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:35.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:34 smithi176 ceph-mon[33619]: pgmap v434: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-04T16:12:35.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]: dispatch 2024-03-04T16:12:35.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:35.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:35.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:35.266 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:35.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:35.396 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:35.472 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-04T16:12:35.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-04T16:12:35.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:35.526 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:35.628 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:35.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:35.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:35.780 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:35.828 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]': finished 2024-03-04T16:12:35.828 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:35 smithi038 ceph-mon[29779]: osdmap e236: 8 total, 8 up, 6 in 2024-03-04T16:12:35.829 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]': finished 2024-03-04T16:12:35.829 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:35 smithi038 ceph-mon[38113]: osdmap e236: 8 total, 8 up, 6 in 2024-03-04T16:12:35.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:35.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:35.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:35.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:12:36.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3668803667' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.7"}]': finished 2024-03-04T16:12:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:35 smithi176 ceph-mon[33619]: osdmap e236: 8 total, 8 up, 6 in 2024-03-04T16:12:36.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.163 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:36.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.290 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:36.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:36.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:36.516 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.540 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:36.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:36.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:36.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.803 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:36.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:36.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:36.929 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:36.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:36 smithi038 ceph-mon[29779]: pgmap v436: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:12:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:36 smithi038 ceph-mon[29779]: osdmap e237: 8 total, 8 up, 6 in 2024-03-04T16:12:36.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:36 smithi038 ceph-mon[38113]: pgmap v436: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:12:36.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:36 smithi038 ceph-mon[38113]: osdmap e237: 8 total, 8 up, 6 in 2024-03-04T16:12:37.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:37.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.055 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:37.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:37.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:36 smithi176 ceph-mon[33619]: pgmap v436: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:12:37.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:36 smithi176 ceph-mon[33619]: osdmap e237: 8 total, 8 up, 6 in 2024-03-04T16:12:37.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:37.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.180 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:37.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:37.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:37.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:37.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.436 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:37.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:37.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:37.562 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:37.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:37.689 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:37.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:37.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:37.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:37.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:37.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:37.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:38.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:38.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:38.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:38.208 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:38.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:38.338 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:38.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:38.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:38.598 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:38 smithi176 ceph-mon[33619]: pgmap v438: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s 2024-03-04T16:12:38.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:38.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.728 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[29779]: pgmap v438: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:38 smithi038 ceph-mon[38113]: pgmap v438: 148 pgs: 148 active+clean; 99 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 859 op/s 2024-03-04T16:12:38.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:38.830 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:38.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:38.983 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:38.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:39.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:39.112 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:39.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:39.214 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:39.242 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:39.344 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.372 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:39.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:39.460 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:39.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:39.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:39.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:39.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:39.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:39.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:39.864 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:39.892 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:39.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:39.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:40.021 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:40.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.148 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:40.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:40.191 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:12:40.191 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:12:40.192 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:12:40.192 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:12:40.192 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:12:40.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.274 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:40.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:40.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:40.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:40.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:40.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:40.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:40.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:40.748 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:12:40.748 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:12:40.749 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-04T16:12:40.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:12:40.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:40.782 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:40.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:40.910 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:40.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:41.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.036 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:41.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:41.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:41.163 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:41.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:41.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:41.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:41.419 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:41.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:41 smithi038 ceph-mon[29779]: pgmap v439: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:12:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:41 smithi038 ceph-mon[38113]: pgmap v439: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:12:41.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:41.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:41 smithi176 ceph-mon[33619]: pgmap v439: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:12:41.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:41.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:41.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:41.806 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:41.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:41.935 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:41.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:42.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:42.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:42.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:42.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:42.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.322 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:42.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:42.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.448 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:42.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:42.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:42.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:42.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.699 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:42.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:42.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:42.828 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:42.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:42.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:42.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:43.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:43.087 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:43.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:43.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:43.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:43.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.216 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:43.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:43.239 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:12:43.317 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:43.345 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:43.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:43.471 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:43.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:43 smithi038 ceph-mon[29779]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 196 op/s 2024-03-04T16:12:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:43.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:43 smithi038 ceph-mon[38113]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 196 op/s 2024-03-04T16:12:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:43 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:43.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:43.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:43.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:43 smithi176 ceph-mon[33619]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 196 op/s 2024-03-04T16:12:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:43 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:12:43.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:43.730 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:43.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:43.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:43.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:43.983 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:43.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:44.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:44.113 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:44.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:44.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:44.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:44.339 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:44.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:12:44.367 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:44.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:44.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:44.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:44 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:44 smithi176 ceph-mon[33619]: osdmap e238: 8 total, 8 up, 6 in 2024-03-04T16:12:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:44.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:44.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:44.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:44.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:44.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[29779]: osdmap e238: 8 total, 8 up, 6 in 2024-03-04T16:12:44.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:44.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:44.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3560490202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:12:44.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:44 smithi038 ceph-mon[38113]: osdmap e238: 8 total, 8 up, 6 in 2024-03-04T16:12:44.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:44.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:44.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:44.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:44.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:45.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:45.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:45.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:45.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.264 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:45.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:45.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:45.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:45.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:45.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:45.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:45 smithi038 ceph-mon[29779]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s 2024-03-04T16:12:45.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:45 smithi038 ceph-mon[38113]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s 2024-03-04T16:12:45.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:45.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:45 smithi176 ceph-mon[33619]: pgmap v442: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s 2024-03-04T16:12:45.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:45.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:45.748 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:45.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:45.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:45.897 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:45.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:45.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:46.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:46.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:46.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:46.199 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:46.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:12:46.249 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:12:46.271 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:12:46.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:46.290 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:12:46.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:46.311 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:12:46.331 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:12:46.352 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:12:46.371 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:12:46.372 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:12:46.373 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:12:46.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:46.432 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:46.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:46 smithi038 ceph-mon[29779]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-04T16:12:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1741305307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:46 smithi038 ceph-mon[38113]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-04T16:12:46.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1741305307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:46.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:46.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:46 smithi176 ceph-mon[33619]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-04T16:12:46.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1741305307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:12:46.662 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:46.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:46.792 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:46.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:46.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:46.947 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:46.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:47.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:47.077 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:47.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:12:47.178 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.235 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:47.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:47.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:47.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:47.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:47.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.495 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:47.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:47.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:47.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:47.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:47.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:47.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.755 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:47.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:47.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:47.881 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:47.881 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:47.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:47.983 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:48.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:48.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:48.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:48.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:48.261 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:48.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.387 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:48.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:48.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:48.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:48.555 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_1' removed 2024-03-04T16:12:48.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:48.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[29779]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 158 op/s 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[38113]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 158 op/s 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:48.698 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:12:48.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:48.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:48 smithi176 ceph-mon[33619]: pgmap v444: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 158 op/s 2024-03-04T16:12:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:48 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:12:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:12:48.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:48.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:48.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:48.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.022 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:49.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:49.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:12:49.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:49.148 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:49.250 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.274 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:49.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:49.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:49.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:49.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:49.514 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-04T16:12:49.515 INFO:tasks.radosbench:Beginning radosbench... 2024-03-04T16:12:49.515 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-04T16:12:49.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-04T16:12:49.525 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:49.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:49.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:49.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:12:49.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[29779]: osdmap e239: 8 total, 8 up, 6 in 2024-03-04T16:12:49.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:49.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:49.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:49.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:12:49.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[38113]: osdmap e239: 8 total, 8 up, 6 in 2024-03-04T16:12:49.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:49.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:49.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:49 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:49.755 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:49.785 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2145128095' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:12:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:49 smithi176 ceph-mon[33619]: osdmap e239: 8 total, 8 up, 6 in 2024-03-04T16:12:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:49 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:12:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:49 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:12:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:49 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:12:49.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:49.912 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:49.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:50.013 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.038 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:50.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:50.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.164 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:50.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:50.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.289 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:50.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:50.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:50.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:50.440 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:50.441 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:50.463 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:50.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:50.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:50.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:50.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:50.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:50.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:50.923 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:50.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:51.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:51.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:51.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:51.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:51.313 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[38113]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:51.313 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:12:51.313 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:12:51.313 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[38113]: osdmap e240: 8 total, 8 up, 6 in 2024-03-04T16:12:51.313 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[29779]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:51.314 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:12:51.314 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:12:51.314 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:51 smithi038 ceph-mon[29779]: osdmap e240: 8 total, 8 up, 6 in 2024-03-04T16:12:51.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:51.431 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:51.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:51.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:51 smithi176 ceph-mon[33619]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:12:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:51 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:12:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:51 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:12:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:51 smithi176 ceph-mon[33619]: osdmap e240: 8 total, 8 up, 6 in 2024-03-04T16:12:51.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:51.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:51.783 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:51.790 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:12:51.790 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:12:51.790 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 240, 2024-03-04T16:12:51.790 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:12:50.566465+0000", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:10:28.206965+0000", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5806080, 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "noscrub", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:12:51.791 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 8, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:12:51.792 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.793 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.794 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "185", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.796 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.797 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.798 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:12:51.799 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.800 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.801 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.802 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:51.804 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6500000953674316, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6500000953674316, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.806 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.807 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.808 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.809 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.810 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.75, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.75, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.309999942779541, 2024-03-04T16:12:51.811 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.309999942779541, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.812 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:51.813 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.814 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.815 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.816 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.817 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5099999904632568, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5099999904632568, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:12:51.818 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:51.819 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.820 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.821 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.821 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.823 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:51.823 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:51.823 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.823 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.823 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.824 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.825 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:51.826 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:12:51.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:12:51.828 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:12:51.829 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:12:51.830 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:12:51.831 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.832 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:12:51.833 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:12:51.834 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 224, 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.835 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:12:51.837 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.838 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 236, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.839 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:12:51.840 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.841 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:12:51.842 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 140, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:12:51.843 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.844 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:12:51.845 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 233, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.847 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:51.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.850 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 209, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.851 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:12:51.852 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.853 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:12:51.854 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.855 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 222, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.856 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.857 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.858 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:12:51.859 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 190, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.860 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.861 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.862 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 218, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:12:51.864 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:51.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:12:51.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.867 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:16.214756+0000", 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:18.787310+0000", 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.869 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:57.712298+0000", 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:49.929744+0000", 2024-03-04T16:12:51.870 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:12.599257+0000", 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.871 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:14.089999+0000", 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.872 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:11:38.936878+0000", 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:12:51.873 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:12:18.722753+0000", 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.13", 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.874 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.1e", 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 3 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-04T16:12:51.875 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:51.876 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:12:51.877 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:12:51.878 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:12:51.879 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:12:51.879 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:12:51.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:51.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:51.981 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:52.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:52.132 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:52.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.258 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:52.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:52.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:52.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:52 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/461635732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:52.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:52 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/461635732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.509 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:52 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/461635732' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:12:52.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:52.734 INFO:tasks.thrashosds.thrasher:j is {'epoch': 240, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:12:50.566465+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:10:28.206965+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '185', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 236, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 233, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 222, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:16.214756+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:18.787310+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:11:57.712298+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:49.929744+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:12.599257+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:14.089999+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:11:38.936878+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:12:18.722753+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '5.13', 'mappings': [{'from': 0, 'to': 5}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:12:52.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:12:52.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:52.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:52.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:52.887 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:52.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:52.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.012 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:53.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:53.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:12:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:12:53.113 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.138 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:53.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:53.239 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:53.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:53.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:53.396 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:53.471 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:53 smithi038 ceph-mon[29779]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:53.471 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:53.472 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:12:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:12:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:12:53.472 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:53 smithi038 ceph-mon[38113]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:53.472 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:53.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.523 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:53.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:53 smithi176 ceph-mon[33619]: pgmap v448: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:12:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:12:53.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.649 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:53.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:53.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:53.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:53.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:53.900 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:53.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:54.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.027 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:54.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:54.128 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.153 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:54.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:54.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:54.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:54.346 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:12:54.346 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:12:54.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:54.347 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:54.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:54.406 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:54.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.535 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:54.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:12:54.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:54.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:54.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:54.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:54.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:54.914 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:54.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:55.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.042 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:55.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:55.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:55.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:55.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:55.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:55.294 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:55.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:55.420 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:55 smithi038 ceph-mon[29779]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:55 smithi038 ceph-mon[38113]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:55.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:55.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:55 smithi176 ceph-mon[33619]: pgmap v449: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:55.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:55.675 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:55.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:55.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:55.801 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:55.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:55.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:55.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:56.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:56.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:56.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:56.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:12:56.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:56.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:56.212 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:12:56.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:56.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:56.411 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:56 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/495770881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.412 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:56 smithi038 ceph-mon[29779]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.412 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:56 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/495770881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.412 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:56 smithi038 ceph-mon[38113]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:56.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:56.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.569 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:56.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:56 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/495770881' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:56 smithi176 ceph-mon[33619]: from='client.35276 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:12:56.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:56.699 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:56.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:56.826 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:56.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:56.956 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:56.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:57.056 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:12:57.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:57.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:57.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:57.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:57.240 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_2' created 2024-03-04T16:12:57.317 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:57.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:57.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:57.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: osdmap e241: 8 total, 8 up, 6 in 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1561324637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[38113]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:12:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: osdmap e241: 8 total, 8 up, 6 in 2024-03-04T16:12:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1561324637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:57 smithi038 ceph-mon[29779]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.604 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: pgmap v450: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 55 KiB/s rd, 54 op/s 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: from='client.35276 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: osdmap e241: 8 total, 8 up, 6 in 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1561324637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:57 smithi176 ceph-mon[33619]: from='client.35282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-04T16:12:57.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:57.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:57.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:57.861 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:57.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:12:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:57.991 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:57.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:12:58.092 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:12:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:58.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:12:58.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:58.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:58.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:58.382 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:58.484 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:58.512 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:58.595 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:58 smithi038 ceph-mon[29779]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-04T16:12:58.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:58 smithi038 ceph-mon[29779]: osdmap e242: 8 total, 8 up, 6 in 2024-03-04T16:12:58.596 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:58 smithi038 ceph-mon[38113]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-04T16:12:58.596 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:58 smithi038 ceph-mon[38113]: osdmap e242: 8 total, 8 up, 6 in 2024-03-04T16:12:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:58 smithi176 ceph-mon[33619]: from='client.35282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-04T16:12:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:58 smithi176 ceph-mon[33619]: osdmap e242: 8 total, 8 up, 6 in 2024-03-04T16:12:58.613 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:12:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.642 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:58.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:12:58.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:58.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.770 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:58.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:58.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:58.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:58.898 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:58.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:58.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:59.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:12:59.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:59.149 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:59.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.275 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:59.376 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:12:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.401 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:12:59.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:12:59.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:59 smithi176 ceph-mon[33619]: pgmap v453: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-03-04T16:12:59.509 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:59 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:12:59.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:59 smithi176 ceph-mon[33619]: osdmap e243: 8 total, 8 up, 6 in 2024-03-04T16:12:59.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:59 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:12:59.510 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:12:59 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:12:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.527 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:12:59.628 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:12:59.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:12:59.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[29779]: pgmap v453: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-03-04T16:12:59.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[29779]: osdmap e243: 8 total, 8 up, 6 in 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[38113]: pgmap v453: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[38113]: osdmap e243: 8 total, 8 up, 6 in 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:12:59.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:12:59 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:12:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:12:59.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:12:59.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-04T16:12:59.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:12:59.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:12:59.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:12:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:12:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:12:59.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:00.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:00.042 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:00.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:00.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:00.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:00.224 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:00.225 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:00.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:00.297 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:00.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:00.422 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:00.439 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:13:00.442 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:13:00.524 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.548 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:00.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:00 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:13:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:00 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:13:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:00 smithi176 ceph-mon[33619]: osdmap e244: 8 total, 8 up, 6 in 2024-03-04T16:13:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:00 smithi176 ceph-mon[33619]: pgmap v456: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:00.649 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[29779]: osdmap e244: 8 total, 8 up, 6 in 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[29779]: pgmap v456: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:13:00.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:13:00.651 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[38113]: osdmap e244: 8 total, 8 up, 6 in 2024-03-04T16:13:00.651 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:00 smithi038 ceph-mon[38113]: pgmap v456: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:00.678 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:00.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:00.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:00.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:00.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:01.033 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:01.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:01.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:01.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.195 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:01.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:01.329 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:01.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:01.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.456 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:01.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:01.490 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:13:01.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:01 smithi038 ceph-mon[29779]: from='client.25279 v1:172.21.15.38:0/771910519' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:13:01.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:01 smithi038 ceph-mon[38113]: from='client.25279 v1:172.21.15.38:0/771910519' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:13:01.509 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:13:01.529 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:13:01.548 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:13:01.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:01.568 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:13:01.585 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:01.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:13:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:01 smithi176 ceph-mon[33619]: from='client.25279 v1:172.21.15.38:0/771910519' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:13:01.608 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:13:01.628 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:13:01.629 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:13:01.630 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:13:01.630 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:13:01.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:01.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:01.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:01.841 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:01.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:01.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:01.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:02.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:02.097 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:02.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:02.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:02.351 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:02.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:02.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:02.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:02 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3601257085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:02.458 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:02 smithi038 ceph-mon[29779]: pgmap v457: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:13:02.458 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:02 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3601257085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:02.458 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:02 smithi038 ceph-mon[38113]: pgmap v457: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:13:02.480 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:02.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:02 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3601257085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:02.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:02 smithi176 ceph-mon[33619]: pgmap v457: 148 pgs: 12 creating+peering, 136 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:13:02.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:02.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:02.735 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:02.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:02.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:02.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:02.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:02.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:03.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:03.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:03.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:03.254 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:03.307 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:03.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:03.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:03.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:03.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:03.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:13:03.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:13:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:13:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:13:03.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:03.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:03 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:13:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:03 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:13:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:03 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:13:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:03 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:13:03.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:03.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:03.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:03.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:03.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:03.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:03.894 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:03.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:03.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:04.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:04.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:04.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:04.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:04.280 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:04.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.410 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:04.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:04.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.539 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:04.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: osdmap e245: 8 total, 8 up, 6 in 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: pgmap v459: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:13:04.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: osdmap e245: 8 total, 8 up, 6 in 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: pgmap v459: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:13:04.575 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:04 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:13:04.577 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-03-04T16:12:59.497502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1955,"num_read_kb":1801,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":952,"ondisk_log_size":952,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":399,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":2334552,"kb_used_data":6720,"kb_used_omap":9,"kb_used_meta":2327734,"kb_avail":560136360,"statfs":{"total":575970213888,"available":573579632640,"internally_reserved":0,"allocated":6881280,"data_stored":4336516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":2383600295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000806"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":383,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236935+0000","last_change":"2024-03-04T16:09:22.868776+0000","last_active":"2024-03-04T16:12:57.236935+0000","last_peered":"2024-03-04T16:12:57.236935+0000","last_clean":"2024-03-04T16:12:57.236935+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:12:57.236935+0000","last_undegraded":"2024-03-04T16:12:57.236935+0000","last_fullsized":"2024-03-04T16:12:57.236935+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.868660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503048+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237207+0000","last_change":"2024-03-04T16:12:19.579690+0000","last_active":"2024-03-04T16:12:57.237207+0000","last_peered":"2024-03-04T16:12:57.237207+0000","last_clean":"2024-03-04T16:12:57.237207+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:12:57.237207+0000","last_undegraded":"2024-03-04T16:12:57.237207+0000","last_fullsized":"2024-03-04T16:12:57.237207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.579557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.579557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:49.119670+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":243,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.240081+0000","last_change":"2024-03-04T16:12:07.441187+0000","last_active":"2024-03-04T16:12:57.240081+0000","last_peered":"2024-03-04T16:12:57.240081+0000","last_clean":"2024-03-04T16:12:57.240081+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:12:57.240081+0000","last_undegraded":"2024-03-04T16:12:57.240081+0000","last_fullsized":"2024-03-04T16:12:57.240081+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:07.441047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:07.441047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:19.732616+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":432,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234915+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:12:57.234915+0000","last_peered":"2024-03-04T16:12:57.234915+0000","last_clean":"2024-03-04T16:12:57.234915+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:12:57.234915+0000","last_undegraded":"2024-03-04T16:12:57.234915+0000","last_fullsized":"2024-03-04T16:12:57.234915+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503179+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":383,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249045+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:12:58.249045+0000","last_peered":"2024-03-04T16:12:58.249045+0000","last_clean":"2024-03-04T16:12:58.249045+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:12:58.249045+0000","last_undegraded":"2024-03-04T16:12:58.249045+0000","last_fullsized":"2024-03-04T16:12:58.249045+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528556+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235068+0000","last_change":"2024-03-04T16:12:14.551847+0000","last_active":"2024-03-04T16:12:57.235068+0000","last_peered":"2024-03-04T16:12:57.235068+0000","last_clean":"2024-03-04T16:12:57.235068+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:12:57.235068+0000","last_undegraded":"2024-03-04T16:12:57.235068+0000","last_fullsized":"2024-03-04T16:12:57.235068+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.551417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.551417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:26.473127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":394,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249033+0000","last_change":"2024-03-04T16:10:29.251726+0000","last_active":"2024-03-04T16:12:58.249033+0000","last_peered":"2024-03-04T16:12:58.249033+0000","last_clean":"2024-03-04T16:12:58.249033+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:12:58.249033+0000","last_undegraded":"2024-03-04T16:12:58.249033+0000","last_fullsized":"2024-03-04T16:12:58.249033+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.752256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":502,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249750+0000","last_change":"2024-03-04T16:12:22.299425+0000","last_active":"2024-03-04T16:12:58.249750+0000","last_peered":"2024-03-04T16:12:58.249750+0000","last_clean":"2024-03-04T16:12:58.249750+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:12:58.249750+0000","last_undegraded":"2024-03-04T16:12:58.249750+0000","last_fullsized":"2024-03-04T16:12:58.249750+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:22.299279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.299279+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:44.689700+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":239,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236567+0000","last_change":"2024-03-04T16:12:22.616322+0000","last_active":"2024-03-04T16:12:57.236567+0000","last_peered":"2024-03-04T16:12:57.236567+0000","last_clean":"2024-03-04T16:12:57.236567+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:12:57.236567+0000","last_undegraded":"2024-03-04T16:12:57.236567+0000","last_fullsized":"2024-03-04T16:12:57.236567+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:22.616197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.616197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:36.166685+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":386,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.232974+0000","last_change":"2024-03-04T16:12:13.585218+0000","last_active":"2024-03-04T16:12:57.232974+0000","last_peered":"2024-03-04T16:12:57.232974+0000","last_clean":"2024-03-04T16:12:57.232974+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:12:57.232974+0000","last_undegraded":"2024-03-04T16:12:57.232974+0000","last_fullsized":"2024-03-04T16:12:57.232974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:13.585073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:12:13.585073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:13.600759+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":408,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234340+0000","last_change":"2024-03-04T16:12:02.625596+0000","last_active":"2024-03-04T16:12:57.234340+0000","last_peered":"2024-03-04T16:12:57.234340+0000","last_clean":"2024-03-04T16:12:57.234340+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:12:57.234340+0000","last_undegraded":"2024-03-04T16:12:57.234340+0000","last_fullsized":"2024-03-04T16:12:57.234340+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.625486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.625486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:09.951055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":452,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237510+0000","last_change":"2024-03-04T16:11:58.878158+0000","last_active":"2024-03-04T16:12:57.237510+0000","last_peered":"2024-03-04T16:12:57.237510+0000","last_clean":"2024-03-04T16:12:57.237510+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:12:57.237510+0000","last_undegraded":"2024-03-04T16:12:57.237510+0000","last_fullsized":"2024-03-04T16:12:57.237510+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:11:58.878047+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:11:58.878047+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:13.371807+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":390,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233383+0000","last_change":"2024-03-04T16:10:02.242059+0000","last_active":"2024-03-04T16:12:57.233383+0000","last_peered":"2024-03-04T16:12:57.233383+0000","last_clean":"2024-03-04T16:12:57.233383+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:12:57.233383+0000","last_undegraded":"2024-03-04T16:12:57.233383+0000","last_fullsized":"2024-03-04T16:12:57.233383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.241922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.241922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845661+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236176+0000","last_change":"2024-03-04T16:12:07.816272+0000","last_active":"2024-03-04T16:12:57.236176+0000","last_peered":"2024-03-04T16:12:57.236176+0000","last_clean":"2024-03-04T16:12:57.236176+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:12:57.236176+0000","last_undegraded":"2024-03-04T16:12:57.236176+0000","last_fullsized":"2024-03-04T16:12:57.236176+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:07.816157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:07.816157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:10.156308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":394,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234120+0000","last_change":"2024-03-04T16:10:00.097245+0000","last_active":"2024-03-04T16:12:57.234120+0000","last_peered":"2024-03-04T16:12:57.234120+0000","last_clean":"2024-03-04T16:12:57.234120+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:12:57.234120+0000","last_undegraded":"2024-03-04T16:12:57.234120+0000","last_fullsized":"2024-03-04T16:12:57.234120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.097027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.097027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":430,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236535+0000","last_change":"2024-03-04T16:12:15.539669+0000","last_active":"2024-03-04T16:12:57.236535+0000","last_peered":"2024-03-04T16:12:57.236535+0000","last_clean":"2024-03-04T16:12:57.236535+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:12:57.236535+0000","last_undegraded":"2024-03-04T16:12:57.236535+0000","last_fullsized":"2024-03-04T16:12:57.236535+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.539528+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:17.585863+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":390,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237065+0000","last_change":"2024-03-04T16:12:20.525886+0000","last_active":"2024-03-04T16:12:57.237065+0000","last_peered":"2024-03-04T16:12:57.237065+0000","last_clean":"2024-03-04T16:12:57.237065+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:12:57.237065+0000","last_undegraded":"2024-03-04T16:12:57.237065+0000","last_fullsized":"2024-03-04T16:12:57.237065+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:20.525761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.525761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:22.692553+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239073+0000","last_change":"2024-03-04T16:08:27.849628+0000","last_active":"2024-03-04T16:12:57.239073+0000","last_peered":"2024-03-04T16:12:57.239073+0000","last_clean":"2024-03-04T16:12:57.239073+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:12:57.239073+0000","last_undegraded":"2024-03-04T16:12:57.239073+0000","last_fullsized":"2024-03-04T16:12:57.239073+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.849443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.849443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677180+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238648+0000","last_change":"2024-03-04T16:12:13.449849+0000","last_active":"2024-03-04T16:12:57.238648+0000","last_peered":"2024-03-04T16:12:57.238648+0000","last_clean":"2024-03-04T16:12:57.238648+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:12:57.238648+0000","last_undegraded":"2024-03-04T16:12:57.238648+0000","last_fullsized":"2024-03-04T16:12:57.238648+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:13.449771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:13.449771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:36.520746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"240'27","reported_seq":193,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238189+0000","last_change":"2024-03-04T16:10:17.515216+0000","last_active":"2024-03-04T16:12:57.238189+0000","last_peered":"2024-03-04T16:12:57.238189+0000","last_clean":"2024-03-04T16:12:57.238189+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:12:57.238189+0000","last_undegraded":"2024-03-04T16:12:57.238189+0000","last_fullsized":"2024-03-04T16:12:57.238189+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:10:01.944401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.944401+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":383,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249886+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:12:58.249886+0000","last_peered":"2024-03-04T16:12:58.249886+0000","last_clean":"2024-03-04T16:12:58.249886+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:12:58.249886+0000","last_undegraded":"2024-03-04T16:12:58.249886+0000","last_fullsized":"2024-03-04T16:12:58.249886+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528718+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238149+0000","last_change":"2024-03-04T16:10:17.459301+0000","last_active":"2024-03-04T16:12:57.238149+0000","last_peered":"2024-03-04T16:12:57.238149+0000","last_clean":"2024-03-04T16:12:57.238149+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:12:57.238149+0000","last_undegraded":"2024-03-04T16:12:57.238149+0000","last_fullsized":"2024-03-04T16:12:57.238149+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.804222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.804222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237558+0000","last_change":"2024-03-04T16:10:17.481021+0000","last_active":"2024-03-04T16:12:57.237558+0000","last_peered":"2024-03-04T16:12:57.237558+0000","last_clean":"2024-03-04T16:12:57.237558+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:12:57.237558+0000","last_undegraded":"2024-03-04T16:12:57.237558+0000","last_fullsized":"2024-03-04T16:12:57.237558+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:23.836451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.836451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":417,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248719+0000","last_change":"2024-03-04T16:08:17.525889+0000","last_active":"2024-03-04T16:12:58.248719+0000","last_peered":"2024-03-04T16:12:58.248719+0000","last_clean":"2024-03-04T16:12:58.248719+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:12:58.248719+0000","last_undegraded":"2024-03-04T16:12:58.248719+0000","last_fullsized":"2024-03-04T16:12:58.248719+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:08:17.525751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:17.525751+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528287+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236551+0000","last_change":"2024-03-04T16:11:09.910387+0000","last_active":"2024-03-04T16:12:57.236551+0000","last_peered":"2024-03-04T16:12:57.236551+0000","last_clean":"2024-03-04T16:12:57.236551+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:12:57.236551+0000","last_undegraded":"2024-03-04T16:12:57.236551+0000","last_fullsized":"2024-03-04T16:12:57.236551+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:11:09.910270+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:11:09.910270+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052489+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":335,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236360+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:12:57.236360+0000","last_peered":"2024-03-04T16:12:57.236360+0000","last_clean":"2024-03-04T16:12:57.236360+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:12:57.236360+0000","last_undegraded":"2024-03-04T16:12:57.236360+0000","last_fullsized":"2024-03-04T16:12:57.236360+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":396,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237467+0000","last_change":"2024-03-04T16:12:12.526320+0000","last_active":"2024-03-04T16:12:57.237467+0000","last_peered":"2024-03-04T16:12:57.237467+0000","last_clean":"2024-03-04T16:12:57.237467+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:12:57.237467+0000","last_undegraded":"2024-03-04T16:12:57.237467+0000","last_fullsized":"2024-03-04T16:12:57.237467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:12:12.526200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:12.526200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:33.722958+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":258,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239864+0000","last_change":"2024-03-04T16:09:58.487155+0000","last_active":"2024-03-04T16:12:57.239864+0000","last_peered":"2024-03-04T16:12:57.239864+0000","last_clean":"2024-03-04T16:12:57.239864+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:12:57.239864+0000","last_undegraded":"2024-03-04T16:12:57.239864+0000","last_fullsized":"2024-03-04T16:12:57.239864+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:58.487024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:58.487024+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807372+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":383,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239148+0000","last_change":"2024-03-04T16:12:23.777697+0000","last_active":"2024-03-04T16:12:57.239148+0000","last_peered":"2024-03-04T16:12:57.239148+0000","last_clean":"2024-03-04T16:12:57.239148+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:12:57.239148+0000","last_undegraded":"2024-03-04T16:12:57.239148+0000","last_fullsized":"2024-03-04T16:12:57.239148+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.777579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.777579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:31.995197+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":29,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248711+0000","last_change":"2024-03-04T16:12:23.623101+0000","last_active":"2024-03-04T16:12:58.248711+0000","last_peered":"2024-03-04T16:12:58.248711+0000","last_clean":"2024-03-04T16:12:58.248711+0000","last_became_active":"2024-03-04T16:12:23.622602+0000","last_became_peered":"2024-03-04T16:12:23.622602+0000","last_unstale":"2024-03-04T16:12:58.248711+0000","last_undegraded":"2024-03-04T16:12:58.248711+0000","last_fullsized":"2024-03-04T16:12:58.248711+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.487604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.487604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:14.618683+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":401,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250202+0000","last_change":"2024-03-04T16:08:27.722863+0000","last_active":"2024-03-04T16:12:58.250202+0000","last_peered":"2024-03-04T16:12:58.250202+0000","last_clean":"2024-03-04T16:12:58.250202+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:12:58.250202+0000","last_undegraded":"2024-03-04T16:12:58.250202+0000","last_fullsized":"2024-03-04T16:12:58.250202+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:27.722599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:27.722599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528375+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":177,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235718+0000","last_change":"2024-03-04T16:12:22.748187+0000","last_active":"2024-03-04T16:12:57.235718+0000","last_peered":"2024-03-04T16:12:57.235718+0000","last_clean":"2024-03-04T16:12:57.235718+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:12:57.235718+0000","last_undegraded":"2024-03-04T16:12:57.235718+0000","last_fullsized":"2024-03-04T16:12:57.235718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:12:22.748049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.748049+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:40.546233+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":397,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236277+0000","last_change":"2024-03-04T16:12:01.877823+0000","last_active":"2024-03-04T16:12:57.236277+0000","last_peered":"2024-03-04T16:12:57.236277+0000","last_clean":"2024-03-04T16:12:57.236277+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:12:57.236277+0000","last_undegraded":"2024-03-04T16:12:57.236277+0000","last_fullsized":"2024-03-04T16:12:57.236277+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:01.877704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:12:01.877704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:16.623732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":107,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235345+0000","last_change":"2024-03-04T16:10:29.238069+0000","last_active":"2024-03-04T16:12:57.235345+0000","last_peered":"2024-03-04T16:12:57.235345+0000","last_clean":"2024-03-04T16:12:57.235345+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:12:57.235345+0000","last_undegraded":"2024-03-04T16:12:57.235345+0000","last_fullsized":"2024-03-04T16:12:57.235345+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.035281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.035281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.502963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.240074+0000","last_change":"2024-03-04T16:12:08.453428+0000","last_active":"2024-03-04T16:12:57.240074+0000","last_peered":"2024-03-04T16:12:57.240074+0000","last_clean":"2024-03-04T16:12:57.240074+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:12:57.240074+0000","last_undegraded":"2024-03-04T16:12:57.240074+0000","last_fullsized":"2024-03-04T16:12:57.240074+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:08.453314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:08.453314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:27.129284+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":409,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250275+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:12:58.250275+0000","last_peered":"2024-03-04T16:12:58.250275+0000","last_clean":"2024-03-04T16:12:58.250275+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:12:58.250275+0000","last_undegraded":"2024-03-04T16:12:58.250275+0000","last_fullsized":"2024-03-04T16:12:58.250275+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":395,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249227+0000","last_change":"2024-03-04T16:12:14.287844+0000","last_active":"2024-03-04T16:12:58.249227+0000","last_peered":"2024-03-04T16:12:58.249227+0000","last_clean":"2024-03-04T16:12:58.249227+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:12:58.249227+0000","last_undegraded":"2024-03-04T16:12:58.249227+0000","last_fullsized":"2024-03-04T16:12:58.249227+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.287598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:15.651897+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236972+0000","last_change":"2024-03-04T16:10:48.950468+0000","last_active":"2024-03-04T16:12:57.236972+0000","last_peered":"2024-03-04T16:12:57.236972+0000","last_clean":"2024-03-04T16:12:57.236972+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:12:57.236972+0000","last_undegraded":"2024-03-04T16:12:57.236972+0000","last_fullsized":"2024-03-04T16:12:57.236972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:48.950175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:48.950175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052385+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":402,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235207+0000","last_change":"2024-03-04T16:12:15.574564+0000","last_active":"2024-03-04T16:12:57.235207+0000","last_peered":"2024-03-04T16:12:57.235207+0000","last_clean":"2024-03-04T16:12:57.235207+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:12:57.235207+0000","last_undegraded":"2024-03-04T16:12:57.235207+0000","last_fullsized":"2024-03-04T16:12:57.235207+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:15.574491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.574491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:23.967866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":434,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249425+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:12:58.249425+0000","last_peered":"2024-03-04T16:12:58.249425+0000","last_clean":"2024-03-04T16:12:58.249425+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:12:58.249425+0000","last_undegraded":"2024-03-04T16:12:58.249425+0000","last_fullsized":"2024-03-04T16:12:58.249425+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236664+0000","last_change":"2024-03-04T16:12:19.525429+0000","last_active":"2024-03-04T16:12:57.236664+0000","last_peered":"2024-03-04T16:12:57.236664+0000","last_clean":"2024-03-04T16:12:57.236664+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:12:57.236664+0000","last_undegraded":"2024-03-04T16:12:57.236664+0000","last_fullsized":"2024-03-04T16:12:57.236664+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.525317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.525317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:48.604560+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":391,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248931+0000","last_change":"2024-03-04T16:12:23.623867+0000","last_active":"2024-03-04T16:12:58.248931+0000","last_peered":"2024-03-04T16:12:58.248931+0000","last_clean":"2024-03-04T16:12:58.248931+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:12:58.248931+0000","last_undegraded":"2024-03-04T16:12:58.248931+0000","last_fullsized":"2024-03-04T16:12:58.248931+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:31.755247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:08:31.755247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237273+0000","last_change":"2024-03-04T16:10:40.920094+0000","last_active":"2024-03-04T16:12:57.237273+0000","last_peered":"2024-03-04T16:12:57.237273+0000","last_clean":"2024-03-04T16:12:57.237273+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:12:57.237273+0000","last_undegraded":"2024-03-04T16:12:57.237273+0000","last_fullsized":"2024-03-04T16:12:57.237273+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:40.919972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:40.919972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052369+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":452,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238246+0000","last_change":"2024-03-04T16:10:29.383816+0000","last_active":"2024-03-04T16:12:57.238246+0000","last_peered":"2024-03-04T16:12:57.238246+0000","last_clean":"2024-03-04T16:12:57.238246+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:12:57.238246+0000","last_undegraded":"2024-03-04T16:12:57.238246+0000","last_fullsized":"2024-03-04T16:12:57.238246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.633480+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677267+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":381,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248513+0000","last_change":"2024-03-04T16:08:09.425155+0000","last_active":"2024-03-04T16:12:58.248513+0000","last_peered":"2024-03-04T16:12:58.248513+0000","last_clean":"2024-03-04T16:12:58.248513+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:12:58.248513+0000","last_undegraded":"2024-03-04T16:12:58.248513+0000","last_fullsized":"2024-03-04T16:12:58.248513+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:09.425046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:09.425046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528347+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238049+0000","last_change":"2024-03-04T16:09:22.656856+0000","last_active":"2024-03-04T16:12:57.238049+0000","last_peered":"2024-03-04T16:12:57.238049+0000","last_clean":"2024-03-04T16:12:57.238049+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:12:57.238049+0000","last_undegraded":"2024-03-04T16:12:57.238049+0000","last_fullsized":"2024-03-04T16:12:57.238049+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.656594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.656594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677231+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238028+0000","last_change":"2024-03-04T16:10:29.247074+0000","last_active":"2024-03-04T16:12:57.238028+0000","last_peered":"2024-03-04T16:12:57.238028+0000","last_clean":"2024-03-04T16:12:57.238028+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:12:57.238028+0000","last_undegraded":"2024-03-04T16:12:57.238028+0000","last_fullsized":"2024-03-04T16:12:57.238028+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.796905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.796905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503087+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":440,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236360+0000","last_change":"2024-03-04T16:10:29.372889+0000","last_active":"2024-03-04T16:12:57.236360+0000","last_peered":"2024-03-04T16:12:57.236360+0000","last_clean":"2024-03-04T16:12:57.236360+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:12:57.236360+0000","last_undegraded":"2024-03-04T16:12:57.236360+0000","last_fullsized":"2024-03-04T16:12:57.236360+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:09:02.785198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:02.785198+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.502919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":264,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237135+0000","last_change":"2024-03-04T16:12:03.904682+0000","last_active":"2024-03-04T16:12:57.237135+0000","last_peered":"2024-03-04T16:12:57.237135+0000","last_clean":"2024-03-04T16:12:57.237135+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:12:57.237135+0000","last_undegraded":"2024-03-04T16:12:57.237135+0000","last_fullsized":"2024-03-04T16:12:57.237135+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:03.903765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.903765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:33.193474+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":539,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234454+0000","last_change":"2024-03-04T16:12:06.530577+0000","last_active":"2024-03-04T16:12:57.234454+0000","last_peered":"2024-03-04T16:12:57.234454+0000","last_clean":"2024-03-04T16:12:57.234454+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:12:57.234454+0000","last_undegraded":"2024-03-04T16:12:57.234454+0000","last_fullsized":"2024-03-04T16:12:57.234454+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:12:06.530482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:06.530482+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:17.622096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":429,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236535+0000","last_change":"2024-03-04T16:11:05.876940+0000","last_active":"2024-03-04T16:12:57.236535+0000","last_peered":"2024-03-04T16:12:57.236535+0000","last_clean":"2024-03-04T16:12:57.236535+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:12:57.236535+0000","last_undegraded":"2024-03-04T16:12:57.236535+0000","last_fullsized":"2024-03-04T16:12:57.236535+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:11:05.876809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:11:05.876809+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052472+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":443,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250484+0000","last_change":"2024-03-04T16:10:17.514015+0000","last_active":"2024-03-04T16:12:58.250484+0000","last_peered":"2024-03-04T16:12:58.250484+0000","last_clean":"2024-03-04T16:12:58.250484+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:12:58.250484+0000","last_undegraded":"2024-03-04T16:12:58.250484+0000","last_fullsized":"2024-03-04T16:12:58.250484+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:17.964877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:17.964877+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":389,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236854+0000","last_change":"2024-03-04T16:12:04.823911+0000","last_active":"2024-03-04T16:12:57.236854+0000","last_peered":"2024-03-04T16:12:57.236854+0000","last_clean":"2024-03-04T16:12:57.236854+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:12:57.236854+0000","last_undegraded":"2024-03-04T16:12:57.236854+0000","last_fullsized":"2024-03-04T16:12:57.236854+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.823773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.823773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:20.084270+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":523,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250764+0000","last_change":"2024-03-04T16:12:20.287137+0000","last_active":"2024-03-04T16:12:58.250764+0000","last_peered":"2024-03-04T16:12:58.250764+0000","last_clean":"2024-03-04T16:12:58.250764+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:12:58.250764+0000","last_undegraded":"2024-03-04T16:12:58.250764+0000","last_fullsized":"2024-03-04T16:12:58.250764+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:12:20.286793+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.286793+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:25.261905+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":288,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235992+0000","last_change":"2024-03-04T16:10:50.000077+0000","last_active":"2024-03-04T16:12:57.235992+0000","last_peered":"2024-03-04T16:12:57.235992+0000","last_clean":"2024-03-04T16:12:57.235992+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:12:57.235992+0000","last_undegraded":"2024-03-04T16:12:57.235992+0000","last_fullsized":"2024-03-04T16:12:57.235992+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:10:49.999919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:49.999919+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052402+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":415,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234746+0000","last_change":"2024-03-04T16:08:58.056799+0000","last_active":"2024-03-04T16:12:57.234746+0000","last_peered":"2024-03-04T16:12:57.234746+0000","last_clean":"2024-03-04T16:12:57.234746+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:12:57.234746+0000","last_undegraded":"2024-03-04T16:12:57.234746+0000","last_fullsized":"2024-03-04T16:12:57.234746+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:08:58.056695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:08:58.056695+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":115,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238229+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:12:57.238229+0000","last_peered":"2024-03-04T16:12:57.238229+0000","last_clean":"2024-03-04T16:12:57.238229+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:12:57.238229+0000","last_undegraded":"2024-03-04T16:12:57.238229+0000","last_fullsized":"2024-03-04T16:12:57.238229+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":381,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249805+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:12:58.249805+0000","last_peered":"2024-03-04T16:12:58.249805+0000","last_clean":"2024-03-04T16:12:58.249805+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:12:58.249805+0000","last_undegraded":"2024-03-04T16:12:58.249805+0000","last_fullsized":"2024-03-04T16:12:58.249805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528653+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235936+0000","last_change":"2024-03-04T16:12:00.921309+0000","last_active":"2024-03-04T16:12:57.235936+0000","last_peered":"2024-03-04T16:12:57.235936+0000","last_clean":"2024-03-04T16:12:57.235936+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:12:57.235936+0000","last_undegraded":"2024-03-04T16:12:57.235936+0000","last_fullsized":"2024-03-04T16:12:57.235936+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.921174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.921174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:21.501195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":250,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239936+0000","last_change":"2024-03-04T16:10:09.625107+0000","last_active":"2024-03-04T16:12:57.239936+0000","last_peered":"2024-03-04T16:12:57.239936+0000","last_clean":"2024-03-04T16:12:57.239936+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:12:57.239936+0000","last_undegraded":"2024-03-04T16:12:57.239936+0000","last_fullsized":"2024-03-04T16:12:57.239936+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:09.624847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:09.624847+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807353+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":397,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237156+0000","last_change":"2024-03-04T16:12:18.923785+0000","last_active":"2024-03-04T16:12:57.237156+0000","last_peered":"2024-03-04T16:12:57.237156+0000","last_clean":"2024-03-04T16:12:57.237156+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:12:57.237156+0000","last_undegraded":"2024-03-04T16:12:57.237156+0000","last_fullsized":"2024-03-04T16:12:57.237156+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:18.923671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.923671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:24.195615+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":386,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235105+0000","last_change":"2024-03-04T16:09:15.043937+0000","last_active":"2024-03-04T16:12:57.235105+0000","last_peered":"2024-03-04T16:12:57.235105+0000","last_clean":"2024-03-04T16:12:57.235105+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:12:57.235105+0000","last_undegraded":"2024-03-04T16:12:57.235105+0000","last_fullsized":"2024-03-04T16:12:57.235105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:09:15.043855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845573+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":293,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236758+0000","last_change":"2024-03-04T16:10:29.242936+0000","last_active":"2024-03-04T16:12:57.236758+0000","last_peered":"2024-03-04T16:12:57.236758+0000","last_clean":"2024-03-04T16:12:57.236758+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:12:57.236758+0000","last_undegraded":"2024-03-04T16:12:57.236758+0000","last_fullsized":"2024-03-04T16:12:57.236758+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.906439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.906439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":406,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249976+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:12:58.249976+0000","last_peered":"2024-03-04T16:12:58.249976+0000","last_clean":"2024-03-04T16:12:58.249976+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:12:58.249976+0000","last_undegraded":"2024-03-04T16:12:58.249976+0000","last_fullsized":"2024-03-04T16:12:58.249976+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528702+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.235400+0000","last_change":"2024-03-04T16:12:57.235400+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.235400+0000","last_undegraded":"2024-03-04T16:12:57.235400+0000","last_fullsized":"2024-03-04T16:12:57.235400+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":388,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233383+0000","last_change":"2024-03-04T16:10:29.344188+0000","last_active":"2024-03-04T16:12:57.233383+0000","last_peered":"2024-03-04T16:12:57.233383+0000","last_clean":"2024-03-04T16:12:57.233383+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:12:57.233383+0000","last_undegraded":"2024-03-04T16:12:57.233383+0000","last_fullsized":"2024-03-04T16:12:57.233383+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:56.044488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:56.044488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":439,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237536+0000","last_change":"2024-03-04T16:12:26.939744+0000","last_active":"2024-03-04T16:12:57.237536+0000","last_peered":"2024-03-04T16:12:57.237536+0000","last_clean":"2024-03-04T16:12:57.237536+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:12:57.237536+0000","last_undegraded":"2024-03-04T16:12:57.237536+0000","last_fullsized":"2024-03-04T16:12:57.237536+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:12:26.939557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.939557+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:55.263205+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":463,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.232731+0000","last_change":"2024-03-04T16:12:03.608848+0000","last_active":"2024-03-04T16:12:57.232731+0000","last_peered":"2024-03-04T16:12:57.232731+0000","last_clean":"2024-03-04T16:12:57.232731+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:12:57.232731+0000","last_undegraded":"2024-03-04T16:12:57.232731+0000","last_fullsized":"2024-03-04T16:12:57.232731+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:03.608639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.608639+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:11.729420+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":528,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236255+0000","last_change":"2024-03-04T16:10:39.929934+0000","last_active":"2024-03-04T16:12:57.236255+0000","last_peered":"2024-03-04T16:12:57.236255+0000","last_clean":"2024-03-04T16:12:57.236255+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:12:57.236255+0000","last_undegraded":"2024-03-04T16:12:57.236255+0000","last_fullsized":"2024-03-04T16:12:57.236255+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:10:39.929809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:39.929809+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052351+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.234874+0000","last_change":"2024-03-04T16:12:57.234874+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.234874+0000","last_undegraded":"2024-03-04T16:12:57.234874+0000","last_fullsized":"2024-03-04T16:12:57.234874+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":377,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249137+0000","last_change":"2024-03-04T16:12:15.261415+0000","last_active":"2024-03-04T16:12:58.249137+0000","last_peered":"2024-03-04T16:12:58.249137+0000","last_clean":"2024-03-04T16:12:58.249137+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:12:58.249137+0000","last_undegraded":"2024-03-04T16:12:58.249137+0000","last_fullsized":"2024-03-04T16:12:58.249137+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:15.261297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.261297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:37.967103+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":403,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237889+0000","last_change":"2024-03-04T16:12:04.845679+0000","last_active":"2024-03-04T16:12:57.237889+0000","last_peered":"2024-03-04T16:12:57.237889+0000","last_clean":"2024-03-04T16:12:57.237889+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:12:57.237889+0000","last_undegraded":"2024-03-04T16:12:57.237889+0000","last_fullsized":"2024-03-04T16:12:57.237889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.845559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.845559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:12.102210+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":423,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235925+0000","last_change":"2024-03-04T16:09:57.809587+0000","last_active":"2024-03-04T16:12:57.235925+0000","last_peered":"2024-03-04T16:12:57.235925+0000","last_clean":"2024-03-04T16:12:57.235925+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:12:57.235925+0000","last_undegraded":"2024-03-04T16:12:57.235925+0000","last_fullsized":"2024-03-04T16:12:57.235925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:09:57.809460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.809460+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503069+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":293,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236469+0000","last_change":"2024-03-04T16:12:18.449606+0000","last_active":"2024-03-04T16:12:57.236469+0000","last_peered":"2024-03-04T16:12:57.236469+0000","last_clean":"2024-03-04T16:12:57.236469+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:12:57.236469+0000","last_undegraded":"2024-03-04T16:12:57.236469+0000","last_fullsized":"2024-03-04T16:12:57.236469+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:18.449487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.449487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:20.099255+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.232056+0000","last_change":"2024-03-04T16:12:57.232056+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.232056+0000","last_undegraded":"2024-03-04T16:12:57.232056+0000","last_fullsized":"2024-03-04T16:12:57.232056+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235757+0000","last_change":"2024-03-04T16:11:59.935508+0000","last_active":"2024-03-04T16:12:57.235757+0000","last_peered":"2024-03-04T16:12:57.235757+0000","last_clean":"2024-03-04T16:12:57.235757+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:12:57.235757+0000","last_undegraded":"2024-03-04T16:12:57.235757+0000","last_fullsized":"2024-03-04T16:12:57.235757+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.935397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.935397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:09.655998+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":115,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238867+0000","last_change":"2024-03-04T16:10:17.466540+0000","last_active":"2024-03-04T16:12:57.238867+0000","last_peered":"2024-03-04T16:12:57.238867+0000","last_clean":"2024-03-04T16:12:57.238867+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:12:57.238867+0000","last_undegraded":"2024-03-04T16:12:57.238867+0000","last_fullsized":"2024-03-04T16:12:57.238867+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:11.936788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:11.936788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":409,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237500+0000","last_change":"2024-03-04T16:12:10.817851+0000","last_active":"2024-03-04T16:12:57.237500+0000","last_peered":"2024-03-04T16:12:57.237500+0000","last_clean":"2024-03-04T16:12:57.237500+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:12:57.237500+0000","last_undegraded":"2024-03-04T16:12:57.237500+0000","last_fullsized":"2024-03-04T16:12:57.237500+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:10.817671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:12:10.817671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:27.214382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"240'35","reported_seq":201,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237883+0000","last_change":"2024-03-04T16:10:29.392036+0000","last_active":"2024-03-04T16:12:57.237883+0000","last_peered":"2024-03-04T16:12:57.237883+0000","last_clean":"2024-03-04T16:12:57.237883+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:12:57.237883+0000","last_undegraded":"2024-03-04T16:12:57.237883+0000","last_fullsized":"2024-03-04T16:12:57.237883+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'19","last_scrub_stamp":"2024-03-04T16:09:23.983090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:23.983090+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.232068+0000","last_change":"2024-03-04T16:12:57.232068+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.232068+0000","last_undegraded":"2024-03-04T16:12:57.232068+0000","last_fullsized":"2024-03-04T16:12:57.232068+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":270,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235443+0000","last_change":"2024-03-04T16:10:47.029618+0000","last_active":"2024-03-04T16:12:57.235443+0000","last_peered":"2024-03-04T16:12:57.235443+0000","last_clean":"2024-03-04T16:12:57.235443+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:12:57.235443+0000","last_undegraded":"2024-03-04T16:12:57.235443+0000","last_fullsized":"2024-03-04T16:12:57.235443+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:47.029491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:10:47.029491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052453+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":115,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237735+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:12:57.237735+0000","last_peered":"2024-03-04T16:12:57.237735+0000","last_clean":"2024-03-04T16:12:57.237735+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:12:57.237735+0000","last_undegraded":"2024-03-04T16:12:57.237735+0000","last_fullsized":"2024-03-04T16:12:57.237735+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":400,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234257+0000","last_change":"2024-03-04T16:10:29.354903+0000","last_active":"2024-03-04T16:12:57.234257+0000","last_peered":"2024-03-04T16:12:57.234257+0000","last_clean":"2024-03-04T16:12:57.234257+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:12:57.234257+0000","last_undegraded":"2024-03-04T16:12:57.234257+0000","last_fullsized":"2024-03-04T16:12:57.234257+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:04.159344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:09:04.159344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":250,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238831+0000","last_change":"2024-03-04T16:10:12.706114+0000","last_active":"2024-03-04T16:12:57.238831+0000","last_peered":"2024-03-04T16:12:57.238831+0000","last_clean":"2024-03-04T16:12:57.238831+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:12:57.238831+0000","last_undegraded":"2024-03-04T16:12:57.238831+0000","last_fullsized":"2024-03-04T16:12:57.238831+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'13","last_scrub_stamp":"2024-03-04T16:10:12.705971+0000","last_deep_scrub":"114'13","last_deep_scrub_stamp":"2024-03-04T16:05:43.834456+0000","last_clean_scrub_stamp":"2024-03-04T16:10:12.705971+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.235025+0000","last_change":"2024-03-04T16:12:57.235025+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.235025+0000","last_undegraded":"2024-03-04T16:12:57.235025+0000","last_fullsized":"2024-03-04T16:12:57.235025+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":377,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250146+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:12:58.250146+0000","last_peered":"2024-03-04T16:12:58.250146+0000","last_clean":"2024-03-04T16:12:58.250146+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:12:58.250146+0000","last_undegraded":"2024-03-04T16:12:58.250146+0000","last_fullsized":"2024-03-04T16:12:58.250146+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528669+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":415,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233075+0000","last_change":"2024-03-04T16:10:29.374989+0000","last_active":"2024-03-04T16:12:57.233075+0000","last_peered":"2024-03-04T16:12:57.233075+0000","last_clean":"2024-03-04T16:12:57.233075+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:12:57.233075+0000","last_undegraded":"2024-03-04T16:12:57.233075+0000","last_fullsized":"2024-03-04T16:12:57.233075+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:09:01.160691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.160691+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":421,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236183+0000","last_change":"2024-03-04T16:12:25.567411+0000","last_active":"2024-03-04T16:12:57.236183+0000","last_peered":"2024-03-04T16:12:57.236183+0000","last_clean":"2024-03-04T16:12:57.236183+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:12:57.236183+0000","last_undegraded":"2024-03-04T16:12:57.236183+0000","last_fullsized":"2024-03-04T16:12:57.236183+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:25.567295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.567295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:27.405223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":636,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238478+0000","last_change":"2024-03-04T16:12:18.618110+0000","last_active":"2024-03-04T16:12:57.238478+0000","last_peered":"2024-03-04T16:12:57.238478+0000","last_clean":"2024-03-04T16:12:57.238478+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:12:57.238478+0000","last_undegraded":"2024-03-04T16:12:57.238478+0000","last_fullsized":"2024-03-04T16:12:57.238478+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'30","last_scrub_stamp":"2024-03-04T16:12:18.617922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.617922+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:30.242245+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":231,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239605+0000","last_change":"2024-03-04T16:12:00.447688+0000","last_active":"2024-03-04T16:12:57.239605+0000","last_peered":"2024-03-04T16:12:57.239605+0000","last_clean":"2024-03-04T16:12:57.239605+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:12:57.239605+0000","last_undegraded":"2024-03-04T16:12:57.239605+0000","last_fullsized":"2024-03-04T16:12:57.239605+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.447554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.447554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:09.520460+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.231988+0000","last_change":"2024-03-04T16:12:57.231988+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.231988+0000","last_undegraded":"2024-03-04T16:12:57.231988+0000","last_fullsized":"2024-03-04T16:12:57.231988+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":389,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237896+0000","last_change":"2024-03-04T16:10:44.996119+0000","last_active":"2024-03-04T16:12:57.237896+0000","last_peered":"2024-03-04T16:12:57.237896+0000","last_clean":"2024-03-04T16:12:57.237896+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:12:57.237896+0000","last_undegraded":"2024-03-04T16:12:57.237896+0000","last_fullsized":"2024-03-04T16:12:57.237896+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:44.995993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:44.995993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052438+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":408,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237269+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:12:57.237269+0000","last_peered":"2024-03-04T16:12:57.237269+0000","last_clean":"2024-03-04T16:12:57.237269+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:12:57.237269+0000","last_undegraded":"2024-03-04T16:12:57.237269+0000","last_fullsized":"2024-03-04T16:12:57.237269+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248729+0000","last_change":"2024-03-04T16:10:17.480211+0000","last_active":"2024-03-04T16:12:58.248729+0000","last_peered":"2024-03-04T16:12:58.248729+0000","last_clean":"2024-03-04T16:12:58.248729+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:12:58.248729+0000","last_undegraded":"2024-03-04T16:12:58.248729+0000","last_fullsized":"2024-03-04T16:12:58.248729+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:21.997653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:09:21.997653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":39,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238518+0000","last_change":"2024-03-04T16:12:35.728287+0000","last_active":"2024-03-04T16:12:57.238518+0000","last_peered":"2024-03-04T16:12:57.238518+0000","last_clean":"2024-03-04T16:12:57.238518+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:12:57.238518+0000","last_undegraded":"2024-03-04T16:12:57.238518+0000","last_fullsized":"2024-03-04T16:12:57.238518+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:11:57.950523+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:11:57.950523+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:18.308155+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":115,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235431+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:12:57.235431+0000","last_peered":"2024-03-04T16:12:57.235431+0000","last_clean":"2024-03-04T16:12:57.235431+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:12:57.235431+0000","last_undegraded":"2024-03-04T16:12:57.235431+0000","last_fullsized":"2024-03-04T16:12:57.235431+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.257007+0000","last_change":"2024-03-04T16:12:58.256227+0000","last_active":"2024-03-04T16:12:58.257007+0000","last_peered":"2024-03-04T16:12:58.257007+0000","last_clean":"2024-03-04T16:12:58.257007+0000","last_became_active":"2024-03-04T16:12:58.255897+0000","last_became_peered":"2024-03-04T16:12:58.255897+0000","last_unstale":"2024-03-04T16:12:58.257007+0000","last_undegraded":"2024-03-04T16:12:58.257007+0000","last_fullsized":"2024-03-04T16:12:58.257007+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:02.486176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":390,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235414+0000","last_change":"2024-03-04T16:12:23.816872+0000","last_active":"2024-03-04T16:12:57.235414+0000","last_peered":"2024-03-04T16:12:57.235414+0000","last_clean":"2024-03-04T16:12:57.235414+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:12:57.235414+0000","last_undegraded":"2024-03-04T16:12:57.235414+0000","last_fullsized":"2024-03-04T16:12:57.235414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.816724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.816724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:44.757733+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":702,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236567+0000","last_change":"2024-03-04T16:10:33.829546+0000","last_active":"2024-03-04T16:12:57.236567+0000","last_peered":"2024-03-04T16:12:57.236567+0000","last_clean":"2024-03-04T16:12:57.236567+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:12:57.236567+0000","last_undegraded":"2024-03-04T16:12:57.236567+0000","last_fullsized":"2024-03-04T16:12:57.236567+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:10:33.829403+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:10:33.829403+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":395,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248569+0000","last_change":"2024-03-04T16:10:29.340288+0000","last_active":"2024-03-04T16:12:58.248569+0000","last_peered":"2024-03-04T16:12:58.248569+0000","last_clean":"2024-03-04T16:12:58.248569+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:12:58.248569+0000","last_undegraded":"2024-03-04T16:12:58.248569+0000","last_fullsized":"2024-03-04T16:12:58.248569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:13.880790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:09:13.880790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528436+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":216,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236897+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:12:57.236897+0000","last_peered":"2024-03-04T16:12:57.236897+0000","last_clean":"2024-03-04T16:12:57.236897+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:12:57.236897+0000","last_undegraded":"2024-03-04T16:12:57.236897+0000","last_fullsized":"2024-03-04T16:12:57.236897+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":401,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236652+0000","last_change":"2024-03-04T16:12:09.805823+0000","last_active":"2024-03-04T16:12:57.236652+0000","last_peered":"2024-03-04T16:12:57.236652+0000","last_clean":"2024-03-04T16:12:57.236652+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:12:57.236652+0000","last_undegraded":"2024-03-04T16:12:57.236652+0000","last_fullsized":"2024-03-04T16:12:57.236652+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:09.805715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:09.805715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:17.859945+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.235002+0000","last_change":"2024-03-04T16:12:57.235002+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.235002+0000","last_undegraded":"2024-03-04T16:12:57.235002+0000","last_fullsized":"2024-03-04T16:12:57.235002+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":391,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238027+0000","last_change":"2024-03-04T16:12:26.602465+0000","last_active":"2024-03-04T16:12:57.238027+0000","last_peered":"2024-03-04T16:12:57.238027+0000","last_clean":"2024-03-04T16:12:57.238027+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:12:57.238027+0000","last_undegraded":"2024-03-04T16:12:57.238027+0000","last_fullsized":"2024-03-04T16:12:57.238027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:26.602350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.602350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:39.109101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":239,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236483+0000","last_change":"2024-03-04T16:12:26.678617+0000","last_active":"2024-03-04T16:12:57.236483+0000","last_peered":"2024-03-04T16:12:57.236483+0000","last_clean":"2024-03-04T16:12:57.236483+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:12:57.236483+0000","last_undegraded":"2024-03-04T16:12:57.236483+0000","last_fullsized":"2024-03-04T16:12:57.236483+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:26.678400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.678400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:48.501031+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1024,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235863+0000","last_change":"2024-03-04T16:10:59.956106+0000","last_active":"2024-03-04T16:12:57.235863+0000","last_peered":"2024-03-04T16:12:57.235863+0000","last_clean":"2024-03-04T16:12:57.235863+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:12:57.235863+0000","last_undegraded":"2024-03-04T16:12:57.235863+0000","last_fullsized":"2024-03-04T16:12:57.235863+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:10:59.955973+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:10:59.955973+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052508+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":607,"num_read_kb":602,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":283,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239751+0000","last_change":"2024-03-04T16:10:17.517202+0000","last_active":"2024-03-04T16:12:57.239751+0000","last_peered":"2024-03-04T16:12:57.239751+0000","last_clean":"2024-03-04T16:12:57.239751+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:12:57.239751+0000","last_undegraded":"2024-03-04T16:12:57.239751+0000","last_fullsized":"2024-03-04T16:12:57.239751+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'12","last_scrub_stamp":"2024-03-04T16:10:00.540297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.540297+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.807392+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":394,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235940+0000","last_change":"2024-03-04T16:09:55.100999+0000","last_active":"2024-03-04T16:12:57.235940+0000","last_peered":"2024-03-04T16:12:57.235940+0000","last_clean":"2024-03-04T16:12:57.235940+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:12:57.235940+0000","last_undegraded":"2024-03-04T16:12:57.235940+0000","last_fullsized":"2024-03-04T16:12:57.235940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.100871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.100871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845676+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.234139+0000","last_change":"2024-03-04T16:12:57.234139+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.234139+0000","last_undegraded":"2024-03-04T16:12:57.234139+0000","last_fullsized":"2024-03-04T16:12:57.234139+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":395,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237307+0000","last_change":"2024-03-04T16:10:29.247615+0000","last_active":"2024-03-04T16:12:57.237307+0000","last_peered":"2024-03-04T16:12:57.237307+0000","last_clean":"2024-03-04T16:12:57.237307+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:12:57.237307+0000","last_undegraded":"2024-03-04T16:12:57.237307+0000","last_fullsized":"2024-03-04T16:12:57.237307+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:07.995029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.995029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677315+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238293+0000","last_change":"2024-03-04T16:10:17.491352+0000","last_active":"2024-03-04T16:12:57.238293+0000","last_peered":"2024-03-04T16:12:57.238293+0000","last_clean":"2024-03-04T16:12:57.238293+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:12:57.238293+0000","last_undegraded":"2024-03-04T16:12:57.238293+0000","last_fullsized":"2024-03-04T16:12:57.238293+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:24.836431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:24.836431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":377,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233099+0000","last_change":"2024-03-04T16:12:17.635817+0000","last_active":"2024-03-04T16:12:57.233099+0000","last_peered":"2024-03-04T16:12:57.233099+0000","last_clean":"2024-03-04T16:12:57.233099+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:12:57.233099+0000","last_undegraded":"2024-03-04T16:12:57.233099+0000","last_fullsized":"2024-03-04T16:12:57.233099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:12:17.635712+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:12:17.635712+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:24.620794+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":107,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233074+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:12:57.233074+0000","last_peered":"2024-03-04T16:12:57.233074+0000","last_clean":"2024-03-04T16:12:57.233074+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:12:57.233074+0000","last_undegraded":"2024-03-04T16:12:57.233074+0000","last_fullsized":"2024-03-04T16:12:57.233074+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.233137+0000","last_change":"2024-03-04T16:12:57.233137+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.233137+0000","last_undegraded":"2024-03-04T16:12:57.233137+0000","last_fullsized":"2024-03-04T16:12:57.233137+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":398,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236701+0000","last_change":"2024-03-04T16:12:23.696188+0000","last_active":"2024-03-04T16:12:57.236701+0000","last_peered":"2024-03-04T16:12:57.236701+0000","last_clean":"2024-03-04T16:12:57.236701+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:12:57.236701+0000","last_undegraded":"2024-03-04T16:12:57.236701+0000","last_fullsized":"2024-03-04T16:12:57.236701+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.696057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.696057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:27.031869+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":406,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234575+0000","last_change":"2024-03-04T16:10:29.368316+0000","last_active":"2024-03-04T16:12:57.234575+0000","last_peered":"2024-03-04T16:12:57.234575+0000","last_clean":"2024-03-04T16:12:57.234575+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:12:57.234575+0000","last_undegraded":"2024-03-04T16:12:57.234575+0000","last_fullsized":"2024-03-04T16:12:57.234575+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:12.013356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:09:12.013356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":426,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238143+0000","last_change":"2024-03-04T16:10:01.014399+0000","last_active":"2024-03-04T16:12:57.238143+0000","last_peered":"2024-03-04T16:12:57.238143+0000","last_clean":"2024-03-04T16:12:57.238143+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:12:57.238143+0000","last_undegraded":"2024-03-04T16:12:57.238143+0000","last_fullsized":"2024-03-04T16:12:57.238143+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'15","last_scrub_stamp":"2024-03-04T16:10:01.014225+0000","last_deep_scrub":"115'15","last_deep_scrub_stamp":"2024-03-04T16:05:50.295256+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.014225+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677284+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":166,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.233833+0000","last_change":"2024-03-04T16:10:29.369889+0000","last_active":"2024-03-04T16:12:57.233833+0000","last_peered":"2024-03-04T16:12:57.233833+0000","last_clean":"2024-03-04T16:12:57.233833+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:12:57.233833+0000","last_undegraded":"2024-03-04T16:12:57.233833+0000","last_fullsized":"2024-03-04T16:12:57.233833+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:10:03.917637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.917637+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.261036+0000","last_change":"2024-03-04T16:12:58.259682+0000","last_active":"2024-03-04T16:12:58.261036+0000","last_peered":"2024-03-04T16:12:58.261036+0000","last_clean":"2024-03-04T16:12:58.261036+0000","last_became_active":"2024-03-04T16:12:58.259080+0000","last_became_peered":"2024-03-04T16:12:58.259080+0000","last_unstale":"2024-03-04T16:12:58.261036+0000","last_undegraded":"2024-03-04T16:12:58.261036+0000","last_fullsized":"2024-03-04T16:12:58.261036+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:18.241663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":403,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236854+0000","last_change":"2024-03-04T16:11:59.019162+0000","last_active":"2024-03-04T16:12:57.236854+0000","last_peered":"2024-03-04T16:12:57.236854+0000","last_clean":"2024-03-04T16:12:57.236854+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:12:57.236854+0000","last_undegraded":"2024-03-04T16:12:57.236854+0000","last_fullsized":"2024-03-04T16:12:57.236854+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.019082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.019082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:18.714345+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":243,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239814+0000","last_change":"2024-03-04T16:12:25.645249+0000","last_active":"2024-03-04T16:12:57.239814+0000","last_peered":"2024-03-04T16:12:57.239814+0000","last_clean":"2024-03-04T16:12:57.239814+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:12:57.239814+0000","last_undegraded":"2024-03-04T16:12:57.239814+0000","last_fullsized":"2024-03-04T16:12:57.239814+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:12:25.645114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.645114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:38.085094+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":462,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235758+0000","last_change":"2024-03-04T16:12:24.566421+0000","last_active":"2024-03-04T16:12:57.235758+0000","last_peered":"2024-03-04T16:12:57.235758+0000","last_clean":"2024-03-04T16:12:57.235758+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:12:57.235758+0000","last_undegraded":"2024-03-04T16:12:57.235758+0000","last_fullsized":"2024-03-04T16:12:57.235758+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:12:24.566235+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.566235+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:31.190644+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":513,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250494+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:12:58.250494+0000","last_peered":"2024-03-04T16:12:58.250494+0000","last_clean":"2024-03-04T16:12:58.250494+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:12:58.250494+0000","last_undegraded":"2024-03-04T16:12:58.250494+0000","last_fullsized":"2024-03-04T16:12:58.250494+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.234992+0000","last_change":"2024-03-04T16:12:57.234992+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.234992+0000","last_undegraded":"2024-03-04T16:12:57.234992+0000","last_fullsized":"2024-03-04T16:12:57.234992+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237048+0000","last_change":"2024-03-04T16:10:29.247041+0000","last_active":"2024-03-04T16:12:57.237048+0000","last_peered":"2024-03-04T16:12:57.237048+0000","last_clean":"2024-03-04T16:12:57.237048+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:12:57.237048+0000","last_undegraded":"2024-03-04T16:12:57.237048+0000","last_fullsized":"2024-03-04T16:12:57.237048+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:05.803381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:05.803381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.502998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":401,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249429+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:12:58.249429+0000","last_peered":"2024-03-04T16:12:58.249429+0000","last_clean":"2024-03-04T16:12:58.249429+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:12:58.249429+0000","last_undegraded":"2024-03-04T16:12:58.249429+0000","last_fullsized":"2024-03-04T16:12:58.249429+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528685+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":492,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250437+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:12:58.250437+0000","last_peered":"2024-03-04T16:12:58.250437+0000","last_clean":"2024-03-04T16:12:58.250437+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:12:58.250437+0000","last_undegraded":"2024-03-04T16:12:58.250437+0000","last_fullsized":"2024-03-04T16:12:58.250437+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528770+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235441+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:12:57.235441+0000","last_peered":"2024-03-04T16:12:57.235441+0000","last_clean":"2024-03-04T16:12:57.235441+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:12:57.235441+0000","last_undegraded":"2024-03-04T16:12:57.235441+0000","last_fullsized":"2024-03-04T16:12:57.235441+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503159+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.260456+0000","last_change":"2024-03-04T16:12:58.259473+0000","last_active":"2024-03-04T16:12:58.260456+0000","last_peered":"2024-03-04T16:12:58.260456+0000","last_clean":"2024-03-04T16:12:58.260456+0000","last_became_active":"2024-03-04T16:12:58.259164+0000","last_became_peered":"2024-03-04T16:12:58.259164+0000","last_unstale":"2024-03-04T16:12:58.260456+0000","last_undegraded":"2024-03-04T16:12:58.260456+0000","last_fullsized":"2024-03-04T16:12:58.260456+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:01.550948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":394,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.234914+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:12:57.234914+0000","last_peered":"2024-03-04T16:12:57.234914+0000","last_clean":"2024-03-04T16:12:57.234914+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:12:57.234914+0000","last_undegraded":"2024-03-04T16:12:57.234914+0000","last_fullsized":"2024-03-04T16:12:57.234914+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845719+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":518,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237757+0000","last_change":"2024-03-04T16:11:59.819002+0000","last_active":"2024-03-04T16:12:57.237757+0000","last_peered":"2024-03-04T16:12:57.237757+0000","last_clean":"2024-03-04T16:12:57.237757+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:12:57.237757+0000","last_undegraded":"2024-03-04T16:12:57.237757+0000","last_fullsized":"2024-03-04T16:12:57.237757+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:11:59.818881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.818881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:12.870905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":389,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250363+0000","last_change":"2024-03-04T16:08:57.714665+0000","last_active":"2024-03-04T16:12:58.250363+0000","last_peered":"2024-03-04T16:12:58.250363+0000","last_clean":"2024-03-04T16:12:58.250363+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:12:58.250363+0000","last_undegraded":"2024-03-04T16:12:58.250363+0000","last_fullsized":"2024-03-04T16:12:58.250363+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:57.714539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:08:57.714539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"240'27","reported_seq":434,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.239451+0000","last_change":"2024-03-04T16:12:24.577754+0000","last_active":"2024-03-04T16:12:57.239451+0000","last_peered":"2024-03-04T16:12:57.239451+0000","last_clean":"2024-03-04T16:12:57.239451+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:12:57.239451+0000","last_undegraded":"2024-03-04T16:12:57.239451+0000","last_fullsized":"2024-03-04T16:12:57.239451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:24.577573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.577573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:34.434658+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.233915+0000","last_change":"2024-03-04T16:12:57.233915+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.233915+0000","last_undegraded":"2024-03-04T16:12:57.233915+0000","last_fullsized":"2024-03-04T16:12:57.233915+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":388,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.250502+0000","last_change":"2024-03-04T16:12:26.381739+0000","last_active":"2024-03-04T16:12:58.250502+0000","last_peered":"2024-03-04T16:12:58.250502+0000","last_clean":"2024-03-04T16:12:58.250502+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:12:58.250502+0000","last_undegraded":"2024-03-04T16:12:58.250502+0000","last_fullsized":"2024-03-04T16:12:58.250502+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:12:26.381198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.381198+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:32.723251+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":392,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235851+0000","last_change":"2024-03-04T16:10:29.344545+0000","last_active":"2024-03-04T16:12:57.235851+0000","last_peered":"2024-03-04T16:12:57.235851+0000","last_clean":"2024-03-04T16:12:57.235851+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:12:57.235851+0000","last_undegraded":"2024-03-04T16:12:57.235851+0000","last_fullsized":"2024-03-04T16:12:57.235851+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.138569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.138569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:57.845616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":393,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.235539+0000","last_change":"2024-03-04T16:10:17.472526+0000","last_active":"2024-03-04T16:12:57.235539+0000","last_peered":"2024-03-04T16:12:57.235539+0000","last_clean":"2024-03-04T16:12:57.235539+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:12:57.235539+0000","last_undegraded":"2024-03-04T16:12:57.235539+0000","last_fullsized":"2024-03-04T16:12:57.235539+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:06.758565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:09:06.758565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.502980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":445,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236876+0000","last_change":"2024-03-04T16:10:29.365556+0000","last_active":"2024-03-04T16:12:57.236876+0000","last_peered":"2024-03-04T16:12:57.236876+0000","last_clean":"2024-03-04T16:12:57.236876+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:12:57.236876+0000","last_undegraded":"2024-03-04T16:12:57.236876+0000","last_fullsized":"2024-03-04T16:12:57.236876+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'16","last_scrub_stamp":"2024-03-04T16:10:03.830634+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.830634+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503142+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":242,"state":"creating+peering","last_fresh":"2024-03-04T16:12:57.233610+0000","last_change":"2024-03-04T16:12:57.233610+0000","last_active":"2024-03-04T16:12:57.228452+0000","last_peered":"2024-03-04T16:12:57.228452+0000","last_clean":"2024-03-04T16:12:57.228452+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-04T16:12:57.233610+0000","last_undegraded":"2024-03-04T16:12:57.233610+0000","last_fullsized":"2024-03-04T16:12:57.233610+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":123,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.249273+0000","last_change":"2024-03-04T16:10:29.323919+0000","last_active":"2024-03-04T16:12:58.249273+0000","last_peered":"2024-03-04T16:12:58.249273+0000","last_clean":"2024-03-04T16:12:58.249273+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:12:58.249273+0000","last_undegraded":"2024-03-04T16:12:58.249273+0000","last_fullsized":"2024-03-04T16:12:58.249273+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:08:30.839981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:08:30.839981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":387,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238583+0000","last_change":"2024-03-04T16:09:01.586219+0000","last_active":"2024-03-04T16:12:57.238583+0000","last_peered":"2024-03-04T16:12:57.238583+0000","last_clean":"2024-03-04T16:12:57.238583+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:12:57.238583+0000","last_undegraded":"2024-03-04T16:12:57.238583+0000","last_fullsized":"2024-03-04T16:12:57.238583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:01.586142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:01.586142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.677215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":107,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237402+0000","last_change":"2024-03-04T16:10:29.322011+0000","last_active":"2024-03-04T16:12:57.237402+0000","last_peered":"2024-03-04T16:12:57.237402+0000","last_clean":"2024-03-04T16:12:57.237402+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:12:57.237402+0000","last_undegraded":"2024-03-04T16:12:57.237402+0000","last_fullsized":"2024-03-04T16:12:57.237402+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:22.995100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:09:22.995100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":488,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.236723+0000","last_change":"2024-03-04T16:10:29.356897+0000","last_active":"2024-03-04T16:12:57.236723+0000","last_peered":"2024-03-04T16:12:57.236723+0000","last_clean":"2024-03-04T16:12:57.236723+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:12:57.236723+0000","last_undegraded":"2024-03-04T16:12:57.236723+0000","last_fullsized":"2024-03-04T16:12:57.236723+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'17","last_scrub_stamp":"2024-03-04T16:09:56.769841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.769841+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.503107+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.260879+0000","last_change":"2024-03-04T16:12:58.259516+0000","last_active":"2024-03-04T16:12:58.260879+0000","last_peered":"2024-03-04T16:12:58.260879+0000","last_clean":"2024-03-04T16:12:58.260879+0000","last_became_active":"2024-03-04T16:12:58.259119+0000","last_became_peered":"2024-03-04T16:12:58.259119+0000","last_unstale":"2024-03-04T16:12:58.260879+0000","last_undegraded":"2024-03-04T16:12:58.260879+0000","last_fullsized":"2024-03-04T16:12:58.260879+0000","mapping_epoch":242,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:19.547937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":389,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237370+0000","last_change":"2024-03-04T16:12:00.783064+0000","last_active":"2024-03-04T16:12:57.237370+0000","last_peered":"2024-03-04T16:12:57.237370+0000","last_clean":"2024-03-04T16:12:57.237370+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:12:57.237370+0000","last_undegraded":"2024-03-04T16:12:57.237370+0000","last_fullsized":"2024-03-04T16:12:57.237370+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.782828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.782828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:27.955404+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":239,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.237389+0000","last_change":"2024-03-04T16:10:38.935575+0000","last_active":"2024-03-04T16:12:57.237389+0000","last_peered":"2024-03-04T16:12:57.237389+0000","last_clean":"2024-03-04T16:12:57.237389+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:12:57.237389+0000","last_undegraded":"2024-03-04T16:12:57.237389+0000","last_fullsized":"2024-03-04T16:12:57.237389+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:10:38.935466+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:10:38.935466+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:12:58.052422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":401,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-04T16:12:58.248838+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:12:58.248838+0000","last_peered":"2024-03-04T16:12:58.248838+0000","last_clean":"2024-03-04T16:12:58.248838+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:12:58.248838+0000","last_undegraded":"2024-03-04T16:12:58.248838+0000","last_fullsized":"2024-03-04T16:12:58.248838+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:01.528734+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":431,"reported_epoch":242,"state":"active+clean","last_fresh":"2024-03-04T16:12:57.238128+0000","last_change":"2024-03-04T16:10:29.336876+0000","last_active":"2024-03-04T16:12:57.238128+0000","last_peered":"2024-03-04T16:12:57.238128+0000","last_clean":"2024-03-04T16:12:57.238128+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:12:57.238128+0000","last_undegraded":"2024-03-04T16:12:57.238128+0000","last_fullsized":"2024-03-04T16:12:57.238128+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'10","last_scrub_stamp":"2024-03-04T16:09:08.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:08.714557+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:13:00.502944+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":607,"num_read_kb":602,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":729,"ondisk_log_size":729,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070591,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325512,"kb_used_data":1404,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93419640,"statfs":{"total":95995035648,"available":95661711360,"internally_reserved":0,"allocated":1437696,"data_stored":1014997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":331872714},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364931,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401896,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343256,"statfs":{"total":95995035648,"available":95583494144,"internally_reserved":0,"allocated":1466368,"data_stored":1021272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339424,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93405728,"statfs":{"total":95995035648,"available":95647465472,"internally_reserved":0,"allocated":1458176,"data_stored":1022167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149842,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":527640,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93217512,"statfs":{"total":95995035648,"available":95454732288,"internally_reserved":0,"allocated":864256,"data_stored":431488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":539425224},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953613,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334664,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410488,"statfs":{"total":95995035648,"available":95652339712,"internally_reserved":0,"allocated":843776,"data_stored":430232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626569,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405416,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":93339736,"statfs":{"total":95995035648,"available":95579889664,"internally_reserved":0,"allocated":811008,"data_stored":416360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":414316998},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:13:04.584 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: osdmap e245: 8 total, 8 up, 6 in 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: pgmap v459: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:13:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:13:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:13:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:13:04.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:04 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:13:04.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.665 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:04.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:04.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:04.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.790 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:04.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:04.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:04.920 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:04.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:05.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:05.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:05.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:05.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:05.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:05.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:05.390 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:05 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:13:05.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:05 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:13:05.391 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:05 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:13:05.391 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:05 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:13:05.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.427 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:05.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:05.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:05.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:05 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:13:05.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:05 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:13:05.636 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:13:05.637 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:13:05.637 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-04T16:13:05.637 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:13:05.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:05.685 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:05.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.813 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:05.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:05.914 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:05.942 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:05.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:06.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:06.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:06.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:06.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:06.303 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:06.331 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:06.343 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:13:06.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.458 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:06.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:06.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:06.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:13:06.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:13:06.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:06.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.717 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:06.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-03-04T16:13:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:13:06.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: pgmap v460: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 705 B/s rd, 0 op/s 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:13:06.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:06.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:13:06.818 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.842 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:06.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:06.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:06.967 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:06.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:07.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:07.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:07.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:13:07.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:07.237 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:07.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:07.365 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:07.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.492 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:07.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:07.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:07.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:13:07.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:07.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:07.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: osdmap e246: 8 total, 8 up, 6 in 2024-03-04T16:13:07.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:13:07.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:07 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:13:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.623 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:07.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:07.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:13:07.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:13:07.626 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: osdmap e246: 8 total, 8 up, 6 in 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:13:07.627 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2244405175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: osdmap e246: 8 total, 8 up, 6 in 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:13:07.628 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:07 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:13:07.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:07.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:07.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:07.876 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:07.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:07.966 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:07.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:08.002 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:08.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:08.129 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:08.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:08.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:08.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:08.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.386 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:08.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:08.510 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:08.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.637 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:08.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:08.766 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:08 smithi176 ceph-mon[33619]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:13:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:08 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:13:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:08 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:13:08.868 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:08.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:08.894 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:08.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[29779]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[38113]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:08.901 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:13:08.902 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:08 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:13:08.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:09.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:09.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:09.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.150 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:09.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:09.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.403 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:09.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:13:09.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:09.580 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-04T16:13:09.585 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:13:09.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 1 2024-03-04T16:13:09.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:09.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:09.784 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2029299092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:09.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:09 smithi176 ceph-mon[33619]: from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:09.886 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:09.915 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:09.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:09.955 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2029299092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:09 smithi038 ceph-mon[38113]: from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2029299092' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:09 smithi038 ceph-mon[29779]: from='client.35306 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:13:10.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.048 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:10.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:10.173 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:10.275 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:10.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:10.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.431 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:10.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:10.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:10.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.561 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:10.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:10.662 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:10.688 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: pgmap v463: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: osdmap e247: 8 total, 8 up, 6 in 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:13:10.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[29779]: osdmap e248: 8 total, 8 up, 6 in 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: pgmap v463: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: osdmap e247: 8 total, 8 up, 6 in 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:13:10.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:10 smithi038 ceph-mon[38113]: osdmap e248: 8 total, 8 up, 6 in 2024-03-04T16:13:10.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:10.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: pgmap v463: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:13:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: from='client.35306 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:13:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: osdmap e247: 8 total, 8 up, 6 in 2024-03-04T16:13:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:13:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:13:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:10 smithi176 ceph-mon[33619]: osdmap e248: 8 total, 8 up, 6 in 2024-03-04T16:13:10.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:10.944 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:10.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:10.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:10.979 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:10.984 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:13:10.984 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:13:10.984 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:13:10.984 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:13:10.984 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:13:11.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:11.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:11.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:11.196 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:11.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:11.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:11.427 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:11.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:11.490 DEBUG:teuthology.orchestra.run.smithi038:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-04T16:13:11.493 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-04T16:13:11.493 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-04T16:13:11.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:hints = 1 2024-03-04T16:13:11.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-04T16:13:11.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object prefix: benchmark_data_smithi038_131582 2024-03-04T16:13:11.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:13:11.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 0 0 0 0 0 0 - 0 2024-03-04T16:13:11.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:11.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:11.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:11.738 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:13:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:11 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:13:11.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:11 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:13:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:11 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:13:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:11 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:13:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:11 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:13:11.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:11.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:11.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:12.010 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:12.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:12.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:12.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:12.273 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:12.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:12.410 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:12.512 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:12.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 1 16 3218 3202 200.113 200.125 0.00590564 0.00498091 2024-03-04T16:13:12.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:12.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.687 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:12.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:12.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:13:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.821 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:12.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: pgmap v466: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:13:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:13:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:13:12.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:13:12.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:13:12.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: pgmap v466: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:13:12.906 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: pgmap v466: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:13:12.907 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:12 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:13:12.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:12.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:12.968 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:12.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:13.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:13.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:13.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:13.254 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.285 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:13.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:13.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:13.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:13.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:13.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:13.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:13.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 2 16 6355 6339 198.079 196.062 0.0107339 0.00503046 2024-03-04T16:13:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:13.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:13.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:13.714 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:13.804 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:13:13.805 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:13 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:13:13.818 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:13.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:13 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:13:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:13 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:13:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:13 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:13:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:13 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:13:13.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:13.989 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:13.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:14.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:14.130 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:14.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:14.278 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:14.380 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:14.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:14.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:14.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 3 15 9255 9240 192.485 181.312 0.0048257 0.00518704 2024-03-04T16:13:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.549 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:14.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:14.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.679 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:14.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[29779]: 6.11 deep-scrub starts 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[29779]: pgmap v467: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.4 KiB/s rd, 41 MiB/s wr, 655 op/s 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[29779]: 6.11 deep-scrub ok 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[29779]: 4.a deep-scrub starts 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[29779]: 4.a deep-scrub ok 2024-03-04T16:13:14.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[38113]: 6.11 deep-scrub starts 2024-03-04T16:13:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[38113]: pgmap v467: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.4 KiB/s rd, 41 MiB/s wr, 655 op/s 2024-03-04T16:13:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[38113]: 6.11 deep-scrub ok 2024-03-04T16:13:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[38113]: 4.a deep-scrub starts 2024-03-04T16:13:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:14 smithi038 ceph-mon[38113]: 4.a deep-scrub ok 2024-03-04T16:13:14.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:14.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:14 smithi176 ceph-mon[33619]: 6.11 deep-scrub starts 2024-03-04T16:13:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:14 smithi176 ceph-mon[33619]: pgmap v467: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 1.4 KiB/s rd, 41 MiB/s wr, 655 op/s 2024-03-04T16:13:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:14 smithi176 ceph-mon[33619]: 6.11 deep-scrub ok 2024-03-04T16:13:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:14 smithi176 ceph-mon[33619]: 4.a deep-scrub starts 2024-03-04T16:13:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:14 smithi176 ceph-mon[33619]: 4.a deep-scrub ok 2024-03-04T16:13:14.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:14.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:14.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:15.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:15.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:15.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:15.231 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:15.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.358 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:15.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:15.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:15.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:15.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 4 16 11571 11555 180.533 144.688 0.00259687 0.0055302 2024-03-04T16:13:15.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:15.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:15.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:15.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:15.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:15.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:15.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:15.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:15.964 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:13:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:13:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:13:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:13:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[38113]: osdmap e249: 8 total, 8 up, 6 in 2024-03-04T16:13:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:13:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:13:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:13:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:13:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:15 smithi038 ceph-mon[29779]: osdmap e249: 8 total, 8 up, 6 in 2024-03-04T16:13:16.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:16.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:15 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:13:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:15 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:13:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:15 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:13:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:15 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:13:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:15 smithi176 ceph-mon[33619]: osdmap e249: 8 total, 8 up, 6 in 2024-03-04T16:13:16.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:16.193 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:16.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:16.321 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:16.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:16.456 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:16.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 5 15 14409 14394 179.911 177.438 0.00866762 0.00555104 2024-03-04T16:13:16.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:16.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:16.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:16.709 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[38113]: pgmap v469: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:13:16.710 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:13:16.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:13:16.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[29779]: pgmap v469: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s 2024-03-04T16:13:16.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:13:16.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:16 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:13:16.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:16.752 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:13:16.794 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:13:16.839 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:13:16.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:16.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:16.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:16.886 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:13:16.928 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:13:16.970 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:13:16.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:17.010 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:17.012 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:13:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:13:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:13:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:13:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:13:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:13:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:13:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:13:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:13:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:16 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:13:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:16 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:13:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:16 smithi176 ceph-mon[33619]: pgmap v469: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s 2024-03-04T16:13:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:16 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:13:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:16 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:13:17.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:17.155 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:17.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:17.296 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:17.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:17.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:17.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:17.539 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 6 16 17121 17105 178.163 169.438 0.00371488 0.00560673 2024-03-04T16:13:17.572 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:17.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:17.711 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 4.2 deep-scrub starts 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 4.2 deep-scrub ok 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3700506894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 4.b deep-scrub starts 2024-03-04T16:13:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[29779]: 4.b deep-scrub ok 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 4.2 deep-scrub starts 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 4.2 deep-scrub ok 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3700506894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 4.b deep-scrub starts 2024-03-04T16:13:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:17 smithi038 ceph-mon[38113]: 4.b deep-scrub ok 2024-03-04T16:13:17.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:17.847 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:17.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:17.983 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:17.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:18.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:18.108 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:13:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:13:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 4.2 deep-scrub starts 2024-03-04T16:13:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 4.2 deep-scrub ok 2024-03-04T16:13:18.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:18.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:18.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3700506894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:18.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 4.b deep-scrub starts 2024-03-04T16:13:18.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:17 smithi176 ceph-mon[33619]: 4.b deep-scrub ok 2024-03-04T16:13:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:18.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:18.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:18.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:18.395 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:18.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.532 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:18.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:18.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 7 16 19621 19605 175.03 156.25 0.00446396 0.0057069 2024-03-04T16:13:18.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.674 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:18.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:18.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.810 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:18.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:18.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:18.950 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:18.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:18.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: pgmap v470: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 645 B/s rd, 37 MiB/s wr, 591 op/s 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[29779]: osdmap e250: 8 total, 8 up, 6 in 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:13:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:13:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:13:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:13:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: pgmap v470: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 645 B/s rd, 37 MiB/s wr, 591 op/s 2024-03-04T16:13:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:18 smithi038 ceph-mon[38113]: osdmap e250: 8 total, 8 up, 6 in 2024-03-04T16:13:19.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:19.091 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: pgmap v470: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 293 MiB data, 2.8 GiB used, 534 GiB / 536 GiB avail; 645 B/s rd, 37 MiB/s wr, 591 op/s 2024-03-04T16:13:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:18 smithi176 ceph-mon[33619]: osdmap e250: 8 total, 8 up, 6 in 2024-03-04T16:13:19.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:19.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:19.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.379 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:19.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:19.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:19.522 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:19.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 8 16 21959 21943 171.41 146.125 0.00617237 0.00582773 2024-03-04T16:13:19.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 4.1c deep-scrub starts 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[29779]: 4.1c deep-scrub ok 2024-03-04T16:13:19.649 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:13:19.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:13:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.659 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:19.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:19.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:19.793 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:19.895 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:19.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:19.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:19.948 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:13:19.948 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:13:19.948 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:13:19.948 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:13:19.948 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 4.1c deep-scrub starts 2024-03-04T16:13:19.949 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:19 smithi038 ceph-mon[38113]: 4.1c deep-scrub ok 2024-03-04T16:13:20.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:20.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:13:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 4.1c deep-scrub starts 2024-03-04T16:13:20.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:19 smithi176 ceph-mon[33619]: 4.1c deep-scrub ok 2024-03-04T16:13:20.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:20.227 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:20.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:20.243 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:13:20.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:20.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:20.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:20.505 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:20.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 9 16 24322 24306 168.773 147.688 0.00352257 0.00591889 2024-03-04T16:13:20.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:20.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:20.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:20.774 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:20.876 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:20.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:20.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:20.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: pgmap v472: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[29779]: osdmap e251: 8 total, 8 up, 6 in 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: pgmap v472: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:13:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:13:20.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:20.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:13:20.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:20 smithi038 ceph-mon[38113]: osdmap e251: 8 total, 8 up, 6 in 2024-03-04T16:13:21.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:21.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:13:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:13:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: pgmap v472: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-04T16:13:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:13:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1513090262' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:13:21.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:20 smithi176 ceph-mon[33619]: osdmap e251: 8 total, 8 up, 6 in 2024-03-04T16:13:21.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:13:21.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:21.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.178 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:21.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:21.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:21.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:21.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:21.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:21.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 10 16 26320 26304 164.382 124.875 0.0053922 0.00607767 2024-03-04T16:13:21.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:21.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:21.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:21.771 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:21.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:21.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:21.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:21.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:13:21.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:13:21.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:21.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:21.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[29779]: osdmap e252: 8 total, 8 up, 6 in 2024-03-04T16:13:21.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:13:21.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:13:21.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:21.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:21.947 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:21 smithi038 ceph-mon[38113]: osdmap e252: 8 total, 8 up, 6 in 2024-03-04T16:13:21.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:22.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:21 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:13:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:21 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:13:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:21 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:21 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:21 smithi176 ceph-mon[33619]: osdmap e252: 8 total, 8 up, 6 in 2024-03-04T16:13:22.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:22.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:22.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:22.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:22.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.444 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:22.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:22.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 11 16 28022 28006 159.108 106.375 0.0051367 0.006276 2024-03-04T16:13:22.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:22.578 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:22.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:22.715 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:22.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:22.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:22.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:22.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:22.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:22.996 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:22.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:23.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:23.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:23.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:23.259 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:23.267 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:23.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:23.418 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:23.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:23.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 12 16 30379 30363 158.124 147.312 0.00640227 0.00631891 2024-03-04T16:13:23.555 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[29779]: pgmap v475: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[38113]: pgmap v475: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:23.573 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:23.574 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:23.574 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:23.574 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:23 smithi176 ceph-mon[33619]: pgmap v475: 148 pgs: 148 active+clean; 1.1 GiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-04T16:13:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:23 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:23 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:23.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:23.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:23.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.829 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:23.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:23.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:23.969 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:23.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:24.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:24.110 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:24.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:24.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:24.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:24.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:24.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:24.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[29779]: osdmap e253: 8 total, 8 up, 6 in 2024-03-04T16:13:24.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:24.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:24 smithi038 ceph-mon[38113]: osdmap e253: 8 total, 8 up, 6 in 2024-03-04T16:13:24.499 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:24.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:24.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:24.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 13 15 32342 32327 155.402 122.75 0.00506802 0.00642854 2024-03-04T16:13:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:24 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:24 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:24.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:24 smithi176 ceph-mon[33619]: osdmap e253: 8 total, 8 up, 6 in 2024-03-04T16:13:24.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:24.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:24.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:24.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:24.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:24.944 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:24.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:24.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:25.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:25.076 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:25.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:25.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:25.289 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T16:13:25.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:25.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:25.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:25.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:25.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 14 16 33977 33961 151.596 102.125 0.0054665 0.00659129 2024-03-04T16:13:25.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[29779]: pgmap v476: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 254 MiB/s wr, 4.07k op/s 2024-03-04T16:13:25.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2092794147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.543 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[29779]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.544 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[38113]: pgmap v476: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 254 MiB/s wr, 4.07k op/s 2024-03-04T16:13:25.544 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2092794147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.544 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:25 smithi038 ceph-mon[38113]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.544 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:25.544 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:25.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:25 smithi176 ceph-mon[33619]: pgmap v476: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.7 KiB/s rd, 254 MiB/s wr, 4.07k op/s 2024-03-04T16:13:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2092794147' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:25 smithi176 ceph-mon[33619]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:13:25.621 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:25.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:25.723 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:25.759 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:25.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:25.903 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:25.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:26.005 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:26.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.044 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:26.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:26.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.185 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:26.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:26.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:26.327 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:26.351 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:26.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.465 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:26.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:26.540 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 15 15 36107 36092 150.368 133.188 0.00503717 0.006645 2024-03-04T16:13:26.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:26.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:26 smithi176 ceph-mon[33619]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:13:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:26 smithi176 ceph-mon[33619]: osdmap e254: 8 total, 8 up, 6 in 2024-03-04T16:13:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:26 smithi176 ceph-mon[33619]: pgmap v479: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 971 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-04T16:13:26.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:26.738 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[29779]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:13:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[29779]: osdmap e254: 8 total, 8 up, 6 in 2024-03-04T16:13:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[29779]: pgmap v479: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 971 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-04T16:13:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[38113]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:13:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[38113]: osdmap e254: 8 total, 8 up, 6 in 2024-03-04T16:13:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:26 smithi038 ceph-mon[38113]: pgmap v479: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 971 B/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-04T16:13:26.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:26.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 1 1 2024-03-04T16:13:26.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:26.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:26.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:27.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:27.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:27.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:27.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:27.284 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:27.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:27.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:13:27.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:27.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:27.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:27.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 16 16 37858 37842 147.806 109.375 0.00221863 0.00674967 2024-03-04T16:13:27.553 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:27.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:27.682 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.696 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:27.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.834 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:27.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:27.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:27.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:28.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:28.076 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:28.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:28.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:28.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:28.395 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:13:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.413 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:28.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:28.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 17 16 39745 39729 146.045 117.938 0.00963797 0.00684214 2024-03-04T16:13:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.556 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:28.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:28.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:28.837 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:28 smithi176 ceph-mon[33619]: pgmap v480: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-04T16:13:28.939 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:28.976 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:28.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:28.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:28 smithi038 ceph-mon[29779]: pgmap v480: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-04T16:13:28.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:28 smithi038 ceph-mon[38113]: pgmap v480: 148 pgs: 148 active+clean; 1.8 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-04T16:13:29.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:29.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:29.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:29.258 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:29.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:29.398 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:29.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:29.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 18 16 41327 41311 143.423 98.875 0.00784319 0.00696729 2024-03-04T16:13:29.544 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:29.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:29.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:29.678 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:29.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:29.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:29.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:29.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:29.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:30.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:30.087 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:30.087 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:30.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:13:30.094 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:30.129 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:13:30.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:13:30.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:30.216 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:13:30.235 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:30.255 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:13:30.297 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:13:30.335 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:13:30.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:30.370 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:30.376 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:13:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:13:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:13:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:13:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:13:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:13:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:13:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:13:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:13:30.472 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:30.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:30.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 19 16 43007 42991 141.4 105 0.00637496 0.00706637 2024-03-04T16:13:30.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:30.651 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:30.743 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:30 smithi176 ceph-mon[33619]: pgmap v481: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-04T16:13:30.743 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2213045623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:30.743 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:30 smithi176 ceph-mon[33619]: osdmap e255: 8 total, 8 up, 6 in 2024-03-04T16:13:30.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:30.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:13:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.805 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:30.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:30.906 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[29779]: pgmap v481: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2213045623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[29779]: osdmap e255: 8 total, 8 up, 6 in 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[38113]: pgmap v481: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2213045623' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:30.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:30 smithi038 ceph-mon[38113]: osdmap e255: 8 total, 8 up, 6 in 2024-03-04T16:13:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:30.953 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:30.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:31.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:31.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:31.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:31.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:31.274 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:31.347 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:31.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:31.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:31.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:13:31.542406+0000 min lat: 0.00128089 max lat: 0.151627 avg lat: 0.00715797 2024-03-04T16:13:31.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:13:31.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 20 16 44687 44671 139.58 105 0.00438499 0.00715797 2024-03-04T16:13:31.544 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:31.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:31.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:31.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:31.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:31.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:31.884 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:31.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:31.971 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:31.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:32.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:32.114 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:32.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:32.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:32.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:32.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:32.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:32.536 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:32.541 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 21 16 46434 46418 138.132 109.188 0.00897291 0.00723439 2024-03-04T16:13:32.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:32.682 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:32.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:32 smithi176 ceph-mon[33619]: pgmap v483: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 645 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-04T16:13:32.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:32.821 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:32.879 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:32 smithi038 ceph-mon[29779]: pgmap v483: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 645 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-04T16:13:32.880 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:32 smithi038 ceph-mon[38113]: pgmap v483: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 645 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-04T16:13:32.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:32.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:32.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:33.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:33.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:33.094 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:33.196 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:33.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:33.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:33.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:33.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:13:33.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:33.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:33.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 22 16 47933 47917 136.112 93.6875 0.00910786 0.00734039 2024-03-04T16:13:33.585 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:33.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:33.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:33.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:33.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.766 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:33.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:33.788 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:13:33.788 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:13:33.788 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:13:33.788 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:13:33.788 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:13:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:33.868 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:33.903 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:33.931 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:33.931 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:34.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.049 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:34.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:34.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:34.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.191 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:34.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:34.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:34.324 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:34.427 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:34.461 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:34.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:13:34.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 23 16 49243 49227 133.753 81.875 0.00539691 0.0074717 2024-03-04T16:13:34.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:34.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:34.632 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T16:13:34.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:34.753 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:34.855 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:34.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:34 smithi176 ceph-mon[33619]: pgmap v484: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:13:34.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:13:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:34.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:34.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:34.925 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:34.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:34 smithi038 ceph-mon[29779]: pgmap v484: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:13:34.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:13:34.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:34 smithi038 ceph-mon[38113]: pgmap v484: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:13:34.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:13:35.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:35.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:35.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:35.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:35.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:35.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:35.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:35.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:35.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:35.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 24 16 50779 50763 132.18 96 0.00812876 0.00756016 2024-03-04T16:13:35.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 3 1 2024-03-04T16:13:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:35.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.648 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:35.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:35.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:35.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:35.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:35.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:35.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:35.842 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:13:35.842 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: osdmap e256: 8 total, 8 up, 6 in 2024-03-04T16:13:35.842 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: osdmap e257: 8 total, 8 up, 6 in 2024-03-04T16:13:35.842 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/178420513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:35.842 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[38113]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: osdmap e256: 8 total, 8 up, 6 in 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: osdmap e257: 8 total, 8 up, 6 in 2024-03-04T16:13:35.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/178420513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:35.844 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:35 smithi038 ceph-mon[29779]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:35.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:35.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:36.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:36.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3705603710' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: osdmap e256: 8 total, 8 up, 6 in 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: osdmap e257: 8 total, 8 up, 6 in 2024-03-04T16:13:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/178420513' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:36.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:35 smithi176 ceph-mon[33619]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:36.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.218 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:36.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:36.274 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:13:36.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:36.356 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:36.458 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.500 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:36.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:36.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 25 16 52348 52332 130.815 98.0625 0.00632852 0.00763482 2024-03-04T16:13:36.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:36.628 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[29779]: pgmap v487: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:36.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:36.736 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:36.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:36 smithi176 ceph-mon[33619]: pgmap v487: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:36.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:36 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:36.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:36 smithi176 ceph-mon[33619]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:36.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:36 smithi176 ceph-mon[33619]: osdmap e258: 8 total, 8 up, 6 in 2024-03-04T16:13:36.780 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:36.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:36.840 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24064 7.1 7:8a6e839d:::benchmark_data_smithi038_77863_object24063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 519.33819905500002, 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12912359700000001, 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.841 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24064 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497476+0000", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497478+0000", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.956e-06 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497544+0000", 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6097000000000001e-05 2024-03-04T16:13:36.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497559+0000", 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4935e-05 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497568+0000", 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9970000000000004e-06 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497637+0000", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8853e-05 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497782+0000", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014473599999999999 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497906+0000", 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000124091 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.845 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499051+0000", 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011451409999999999 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499189+0000", 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013806699999999999 2024-03-04T16:13:36.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626465+0000", 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12727575699999999 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626544+0000", 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.907e-05 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626600+0000", 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5896999999999997e-05 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24066 7.1 7:873d823e:::benchmark_data_smithi038_77863_object24065:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:13:36.848 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 519.33652947300004, 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.130573199, 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24066 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:13:36.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499146+0000", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499147+0000", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.274e-06 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499199+0000", 2024-03-04T16:13:36.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2524000000000001e-05 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499208+0000", 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6740000000000003e-06 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499214+0000", 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5999999999999997e-06 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499248+0000", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4187999999999999e-05 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499344+0000", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6173000000000002e-05 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499488+0000", 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014452200000000001 2024-03-04T16:13:36.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.500600+0000", 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011114740000000001 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.501582+0000", 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00098159099999999997 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.853 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629612+0000", 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12803046100000001 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629673+0000", 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1291000000000007e-05 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629719+0000", 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5426999999999997e-05 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.854 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24076 7.1 7:802e523e:::benchmark_data_smithi038_77863_object24075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 519.33175677600002, 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12599276100000001, 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24076 2024-03-04T16:13:36.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503918+0000", 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.503920+0000", 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4190000000000001e-06 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504014+0000", 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4440999999999999e-05 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504025+0000", 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0706999999999999e-05 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504032+0000", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.002e-06 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504086+0000", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4079999999999997e-05 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504200+0000", 2024-03-04T16:13:36.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114207 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.504319+0000", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011859899999999999 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505150+0000", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00083150499999999996 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.505431+0000", 2024-03-04T16:13:36.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028036800000000002 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629771+0000", 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12434076600000001 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629841+0000", 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9821999999999997e-05 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629911+0000", 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9845000000000006e-05 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24080 7.1 7:85c22b13:::benchmark_data_smithi038_77863_object24079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 519.32497828999999, 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11934697599999999, 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24080 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510697+0000", 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510699+0000", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0370000000000001e-06 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510740+0000", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1344000000000003e-05 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510753+0000", 2024-03-04T16:13:36.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3010000000000001e-05 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510760+0000", 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3000000000000004e-06 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510820+0000", 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9786e-05 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.510959+0000", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013839899999999999 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511076+0000", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000117656 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.511948+0000", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00087144999999999996 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.512095+0000", 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000147682 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.629957+0000", 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11786121500000001 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630007+0000", 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0105000000000002e-05 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.630044+0000", 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6992000000000001e-05 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83920 7.1 7:8d8feee9:::benchmark_data_smithi038_77863_object83919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 476.18245648800001, 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10767995700000001, 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.867 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83920 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653219+0000", 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653220+0000", 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.545e-06 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653270+0000", 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9568999999999999e-05 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653281+0000", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1192999999999999e-05 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653296+0000", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4987999999999999e-05 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653345+0000", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8775e-05 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653480+0000", 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013508899999999999 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.653647+0000", 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000167004 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656243+0000", 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025961299999999999 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656442+0000", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000199464 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760825+0000", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.104383084 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760863+0000", 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7704999999999999e-05 2024-03-04T16:13:36.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.760898+0000", 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5410999999999997e-05 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83925 7.1 7:87a9c3a2:::benchmark_data_smithi038_77863_object83924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 476.17910508699998, 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.107333128, 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.874 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83925 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656570+0000", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656572+0000", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.778e-06 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656615+0000", 2024-03-04T16:13:36.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3161000000000002e-05 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656626+0000", 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1065e-05 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656632+0000", 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7389999999999996e-06 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656674+0000", 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2052999999999999e-05 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656764+0000", 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0019999999999995e-05 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.656883+0000", 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119466 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659071+0000", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021878219999999999 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.659140+0000", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8551000000000001e-05 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763805+0000", 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10466587199999999 2024-03-04T16:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763859+0000", 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3298999999999999e-05 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.763903+0000", 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4301999999999997e-05 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84327 7.1 7:8c4037d9:::benchmark_data_smithi038_77863_object84326:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 475.64125207000001, 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15446056899999999, 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84327 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194423+0000", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194425+0000", 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4880000000000001e-06 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194465+0000", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9663000000000003e-05 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194479+0000", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4025999999999999e-05 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194488+0000", 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8810000000000002e-06 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194551+0000", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2869999999999999e-05 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194677+0000", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012619000000000001 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.194823+0000", 2024-03-04T16:13:36.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000145477 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197906+0000", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030836819999999999 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.197926+0000", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9355999999999999e-05 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348763+0000", 2024-03-04T16:13:36.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15083737699999999 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348834+0000", 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1464999999999994e-05 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348883+0000", 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9094000000000001e-05 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84339 7.1 7:81128b26:::benchmark_data_smithi038_77863_object84338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 475.61965786899998, 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132991251, 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84339 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216017+0000", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216020+0000", 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6180000000000002e-06 2024-03-04T16:13:36.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216053+0000", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3401000000000001e-05 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216067+0000", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3816e-05 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.216075+0000", 2024-03-04T16:13:36.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6839999999999996e-06 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217281+0000", 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001206181 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217443+0000", 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000162063 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.217620+0000", 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017687900000000001 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219824+0000", 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022046380000000001 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.219965+0000", 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014102400000000001 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348936+0000", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128970469 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348971+0000", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5068999999999999e-05 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.349008+0000", 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7409e-05 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.892 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163755 7.4 7:20abbf33:::benchmark_data_smithi038_77863_object45487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 391.62392752, 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1434231819999998, 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163755 2024-03-04T16:13:36.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211749+0000", 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4980000000000001e-06 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211755+0000", 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9410000000000004e-06 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211767+0000", 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1882000000000001e-05 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211775+0000", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9910000000000002e-06 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211836+0000", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1597e-05 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211846+0000", 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6569999999999992e-06 2024-03-04T16:13:36.896 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211462+0000", 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99961569800000005 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211477+0000", 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4766999999999999e-05 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.897 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.337886+0000", 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1264093390000001 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339470+0000", 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015841239999999999 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339721+0000", 2024-03-04T16:13:36.898 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025078399999999999 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351914+0000", 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012193641 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351962+0000", 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8062000000000002e-05 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.899 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352023+0000", 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0387999999999998e-05 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355003+0000", 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029802489999999999 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.900 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355055+0000", 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2178000000000001e-05 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355171+0000", 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115386 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.901 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163761 7.4 7:2ccd0a78:::benchmark_data_smithi038_77863_object45493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 391.60808985800003, 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1332310689999998, 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163761 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.902 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227587+0000", 2024-03-04T16:13:36.903 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.812e-06 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227592+0000", 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227603+0000", 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0251000000000001e-05 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.904 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227610+0000", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9650000000000002e-06 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227662+0000", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2834999999999998e-05 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227672+0000", 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8989999999999996e-06 2024-03-04T16:13:36.905 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211510+0000", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98383736899999996 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211518+0000", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5059999999999995e-06 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352101+0000", 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.140582808 2024-03-04T16:13:36.906 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352222+0000", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120686 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352361+0000", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139345 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355375+0000", 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030134020000000001 2024-03-04T16:13:36.907 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.356429+0000", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001054234 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.358238+0000", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018089410000000001 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360641+0000", 2024-03-04T16:13:36.908 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024032419999999999 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360703+0000", 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2532000000000004e-05 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360816+0000", 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011275100000000001 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.909 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:177490 7.1 7:82ecc071:::benchmark_data_smithi038_77863_object59222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 374.37186787600001, 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10186015399999999, 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177490 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.910 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.911 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463809+0000", 2024-03-04T16:13:36.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:36.916 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.767e-06 2024-03-04T16:13:36.916 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463814+0000", 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463824+0000", 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6409999999999997e-06 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.917 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463831+0000", 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1049999999999997e-06 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465019+0000", 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011877140000000001 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465136+0000", 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011716400000000001 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.918 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465249+0000", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011281399999999999 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.468002+0000", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027534370000000001 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.469006+0000", 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010038709999999999 2024-03-04T16:13:36.919 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565552+0000", 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.096545858999999998 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565616+0000", 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4599999999999998e-05 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.920 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565667+0000", 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0862999999999997e-05 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:222378 7.1 7:83b38929:::benchmark_data_smithi038_77863_object104110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 322.73668260400001, 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090076495000000006, 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:13:36.921 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 222378 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.922 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098995+0000", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6989999999999999e-06 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099001+0000", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9739999999999999e-06 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099026+0000", 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4769999999999998e-05 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.923 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099034+0000", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0760000000000003e-06 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099091+0000", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6709000000000002e-05 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099742+0000", 2024-03-04T16:13:36.924 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00065097099999999997 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099852+0000", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110408 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102653+0000", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028013769999999999 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102881+0000", 2024-03-04T16:13:36.925 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227686 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.188963+0000", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.086082045999999995 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189019+0000", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6125000000000002e-05 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189069+0000", 2024-03-04T16:13:36.926 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9654000000000003e-05 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93725 8.2 8:4602f712:::benchmark_data_smithi038_107057_object93724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 210.606036023, 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12168869, 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.927 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93725 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.928 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229641+0000", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5799999999999999e-06 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229692+0000", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1480999999999997e-05 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229701+0000", 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4099999999999997e-06 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.929 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229714+0000", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2140999999999999e-05 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229749+0000", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5058000000000002e-05 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229848+0000", 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9230999999999999e-05 2024-03-04T16:13:36.930 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229968+0000", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119874 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.234094+0000", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041263849999999998 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237005+0000", 2024-03-04T16:13:36.931 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029105889999999999 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351181+0000", 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114175783 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351277+0000", 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6447999999999995e-05 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.932 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351328+0000", 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0710000000000001e-05 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93728 8.2 8:447b8219:::benchmark_data_smithi038_107057_object93727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 210.60503004700001, 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12084650500000001, 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.933 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93728 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:13:36.934 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230646+0000", 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1030000000000001e-06 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230704+0000", 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8165999999999998e-05 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.935 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230714+0000", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9469999999999998e-06 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230721+0000", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1579999999999999e-06 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.231787+0000", 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010659899999999999 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.936 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.231921+0000", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013319300000000001 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.232063+0000", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014256199999999999 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.234204+0000", 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002141034 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.937 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.239004+0000", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004799902 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351370+0000", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11236549999999999 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351455+0000", 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5463999999999994e-05 2024-03-04T16:13:36.938 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351491+0000", 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6486000000000003e-05 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93733 8.2 8:45cfc703:::benchmark_data_smithi038_107057_object93732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 210.59869253700001, 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114652037, 2024-03-04T16:13:36.939 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93733 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.940 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236984+0000", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5939999999999999e-06 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237039+0000", 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5034999999999997e-05 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.941 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237046+0000", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0469999999999996e-06 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237051+0000", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4569999999999998e-06 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237077+0000", 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6139000000000002e-05 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.942 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237169+0000", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2542999999999998e-05 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237262+0000", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3041999999999995e-05 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.239083+0000", 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001821079 2024-03-04T16:13:36.943 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.241678+0000", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002594963 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351528+0000", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.109849579 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351601+0000", 2024-03-04T16:13:36.944 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3014000000000003e-05 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351634+0000", 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3544999999999998e-05 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:114775 8.5 8:aa81b258:::benchmark_data_smithi038_107057_object114774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 188.58847247899999, 2024-03-04T16:13:36.945 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1283407419999998, 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 114775 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.946 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247205+0000", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.249e-06 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247248+0000", 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3449999999999999e-05 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.947 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247261+0000", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2761e-05 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247271+0000", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6320000000000003e-06 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247347+0000", 2024-03-04T16:13:36.948 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6029e-05 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247358+0000", 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1127999999999999e-05 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.232977+0000", 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98561953400000002 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.949 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.232991+0000", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3873000000000001e-05 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.350151+0000", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1171602970000001 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.351593+0000", 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014410849999999999 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.950 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,6,7", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.351847+0000", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025436700000000003 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.364523+0000", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012676031000000001 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.364591+0000", 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8526000000000001e-05 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.951 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.367558+0000", 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029666889999999998 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375350+0000", 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007791765 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.952 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375422+0000", 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1966999999999997e-05 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375543+0000", 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121359 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1839 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93955426377600 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:13:36.953 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 187.977055761, 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.47448410200000002, 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1839 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.954 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858620+0000", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.221e-06 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858625+0000", 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5979999999999999e-06 2024-03-04T16:13:36.955 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858634+0000", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0040000000000005e-06 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858711+0000", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6966999999999997e-05 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858832+0000", 2024-03-04T16:13:36.956 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121279 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858842+0000", 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8330000000000006e-06 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.322095+0000", 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.46325266599999998 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.957 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.322485+0000", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039060800000000003 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.326605+0000", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041195490000000001 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.326845+0000", 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024025300000000001 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.958 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.333103+0000", 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062581240000000003 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1841 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93955430886784 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 187.97687799299999, 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.476828576, 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.959 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1841 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.960 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858798+0000", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7000000000000003e-07 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858802+0000", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8349999999999997e-06 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.961 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858809+0000", 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7200000000000006e-06 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858815+0000", 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5500000000000002e-06 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858864+0000", 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8927000000000001e-05 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.962 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858871+0000", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3069999999999997e-06 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.329016+0000", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.470145074 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.329355+0000", 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000338573 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.963 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.331135+0000", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017796419999999999 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.331607+0000", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000472325 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.335626+0000", 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040187529999999999 2024-03-04T16:13:36.964 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:115508 8.5 8:a9a17dc4:::benchmark_data_smithi038_107057_object115507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.69591008200001, 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080985018000000006, 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 115508 2024-03-04T16:13:36.965 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.966 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139767+0000", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7940000000000001e-06 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139795+0000", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8564999999999998e-05 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139803+0000", 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7330000000000003e-06 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139810+0000", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5060000000000001e-06 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139904+0000", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4498000000000002e-05 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140045+0000", 2024-03-04T16:13:36.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014108400000000001 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,6,7", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.141168+0000", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001123209 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147773+0000", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006605026 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147801+0000", 2024-03-04T16:13:36.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7906e-05 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147809+0000", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2300000000000008e-06 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220629+0000", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072819350000000005 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220666+0000", 2024-03-04T16:13:36.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7020999999999997e-05 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220750+0000", 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4096000000000003e-05 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:115509 8.2 8:447c5fda:::benchmark_data_smithi038_107057_object115508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:13:36.971 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.695851804, 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080756245000000004, 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 115509 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:13:36.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139824+0000", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8599999999999997e-07 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139885+0000", 2024-03-04T16:13:36.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0838999999999998e-05 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139891+0000", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0900000000000001e-06 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139897+0000", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3960000000000003e-06 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139960+0000", 2024-03-04T16:13:36.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2432000000000001e-05 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140080+0000", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012054500000000001 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140227+0000", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014703200000000001 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.145807+0000", 2024-03-04T16:13:36.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005580018 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.148427+0000", 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002619466 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220459+0000", 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072032465000000004 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:13:36.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220525+0000", 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5420999999999996e-05 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220579+0000", 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4654999999999998e-05 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:13:36.977 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:13:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[29779]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:36.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[29779]: osdmap e258: 8 total, 8 up, 6 in 2024-03-04T16:13:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[38113]: pgmap v487: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[38113]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:36.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:36 smithi038 ceph-mon[38113]: osdmap e258: 8 total, 8 up, 6 in 2024-03-04T16:13:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:37.121 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:37.163 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:37.166 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:37.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:37.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:37.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:37.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:37.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 26 16 53950 53934 129.634 100.125 0.00568537 0.00770857 2024-03-04T16:13:37.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:37.585 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:37.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:37.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:37.726 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:37 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:37.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:37 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:37.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:37.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:13:37.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:37.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:37.907 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:37 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:37.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.018 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:38.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:38.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:38.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:38.264 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:38.301 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:38.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:38.542 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 27 16 55473 55457 128.358 95.1875 0.0123381 0.00778514 2024-03-04T16:13:38.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:38.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:38.657 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:38 smithi176 ceph-mon[33619]: pgmap v489: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:38.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:38.733 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:38.757 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:38 smithi038 ceph-mon[29779]: pgmap v489: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:38.757 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:38 smithi038 ceph-mon[38113]: pgmap v489: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-04T16:13:38.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:38.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:38.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:38.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:39.013 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:39.115 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:39.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:39.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:39.285 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:39.360 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:13:39.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:39.404 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:13:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.418 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:39.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:39.445 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:13:39.488 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:13:39.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:39.530 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:13:39.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 28 16 57059 57043 127.311 99.125 0.0133255 0.00784906 2024-03-04T16:13:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:39.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:39.569 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:13:39.609 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:13:39.654 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:13:39.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:13:39.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:13:39.660 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:13:39.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:13:39.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:13:39.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:13:39.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:13:39.661 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:13:39.662 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:39 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2223377509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:39.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:39 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2223377509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:39.662 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:39.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:13:39.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:39.802 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:39 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2223377509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:39.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:39.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:39.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:39.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:40.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.151 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:40.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:40.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:40.284 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:40.284 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:40.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:40.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:40.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:40.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:40.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 29 16 58439 58423 125.895 86.25 0.0143009 0.00793647 2024-03-04T16:13:40.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.584 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:40.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:40.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:40.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:40.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:40.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:40.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:40.951 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:40 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:13:40.951 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:40 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:13:40.951 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:40 smithi176 ceph-mon[33619]: pgmap v490: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:13:40.952 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:40 smithi176 ceph-mon[33619]: osdmap e259: 8 total, 8 up, 6 in 2024-03-04T16:13:40.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:40.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:13:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[29779]: pgmap v490: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[29779]: osdmap e259: 8 total, 8 up, 6 in 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[38113]: pgmap v490: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:13:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:40 smithi038 ceph-mon[38113]: osdmap e259: 8 total, 8 up, 6 in 2024-03-04T16:13:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.005 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:41.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:41.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:41.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:41.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:41.247 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.298 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:41.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:41.369 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:41.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:41.438 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:41.540 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:41.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 30 16 59885 59869 124.711 90.375 0.00845797 0.00801328 2024-03-04T16:13:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.583 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:41.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:41.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:41.717 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:41.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:41.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:41.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:41.995 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:41.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:42.053 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:42.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:42.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:42.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:42.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:42.278 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:42.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:42.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:42.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:42.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 31 16 61384 61368 123.71 93.6875 0.00826859 0.00807892 2024-03-04T16:13:42.555 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:42.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:42.698 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T16:13:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.701 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:42.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[29779]: pgmap v492: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 816 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1084491374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[29779]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[38113]: pgmap v492: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 816 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1084491374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:42 smithi038 ceph-mon[38113]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:42.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:42.906 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:42 smithi176 ceph-mon[33619]: pgmap v492: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 816 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:13:42.906 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:42 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1084491374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.906 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:42 smithi176 ceph-mon[33619]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:13:42.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:42.983 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:42.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:43.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:43.114 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:43.165 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:43.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:43.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:43.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:43.379 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:43.379 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:43.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:43.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:43.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:43.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:43.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 32 16 62869 62853 122.744 92.8125 0.0117999 0.00814235 2024-03-04T16:13:43.636 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:43.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:43.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:43.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:43.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:43.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:43.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[29779]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[29779]: osdmap e260: 8 total, 8 up, 6 in 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[38113]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[38113]: osdmap e260: 8 total, 8 up, 6 in 2024-03-04T16:13:43.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:44.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:13:44.056 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:44.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:43 smithi176 ceph-mon[33619]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:13:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:43 smithi176 ceph-mon[33619]: osdmap e260: 8 total, 8 up, 6 in 2024-03-04T16:13:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:44.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:44.213 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:44.315 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:44.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.349 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:44.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:44.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:44.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:44.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 33 16 63967 63951 121.104 68.625 0.022821 0.0082522 2024-03-04T16:13:44.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:44.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:13:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:44.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:44.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:44.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.778 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:44.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:44 smithi176 ceph-mon[33619]: pgmap v494: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:13:44.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:44.923 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:44.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:44.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:44 smithi038 ceph-mon[29779]: pgmap v494: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:13:44.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:44 smithi038 ceph-mon[38113]: pgmap v494: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.4 KiB/s rd, 113 MiB/s wr, 1.81k op/s 2024-03-04T16:13:45.001 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:45.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:45.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:45.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:45.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:45.307 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:45.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:45.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:45.473 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:45.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 34 16 65135 65119 119.689 73 0.00922064 0.00834966 2024-03-04T16:13:45.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.617 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:45.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:45.719 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:45.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:45.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:45.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:45.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:45.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:45.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.011 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:46.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:46.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:46.149 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:46.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.295 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:46.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:46.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:46.435 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:46.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:46.543 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:46 smithi176 ceph-mon[33619]: osdmap e261: 8 total, 8 up, 6 in 2024-03-04T16:13:46.543 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 35 16 66457 66441 118.63 82.625 0.00670502 0.00841932 2024-03-04T16:13:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.575 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:46.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:46.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:46 smithi038 ceph-mon[29779]: osdmap e261: 8 total, 8 up, 6 in 2024-03-04T16:13:46.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:46 smithi038 ceph-mon[38113]: osdmap e261: 8 total, 8 up, 6 in 2024-03-04T16:13:46.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:46.709 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:46.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:46.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 4 1 2024-03-04T16:13:46.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:46.845 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:46.883 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:46.884 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:46.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:46.974 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:46.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:47.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:47.117 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:47.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:47.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:47.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:47.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:47.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:47.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:47.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:47.524 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:47.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:47 smithi176 ceph-mon[33619]: pgmap v496: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-04T16:13:47.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 36 16 67670 67654 117.44 75.8125 0.00763315 0.0085101 2024-03-04T16:13:47.589 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:13:47.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.684 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:47.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:47 smithi038 ceph-mon[29779]: pgmap v496: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-04T16:13:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:47 smithi038 ceph-mon[38113]: pgmap v496: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-04T16:13:47.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.825 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:47.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:47.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:47.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:47.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:47.973 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:48.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:48.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:48.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:48.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:48.332 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:48.367 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:48.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:48.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:48.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:48.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 37 16 69006 68990 116.523 83.5 0.00440178 0.00857708 2024-03-04T16:13:48.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.652 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:48.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:48.753 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.791 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:48.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:48.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:48.936 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:48.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:49.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:49.068 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:49.090 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:49.091 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:49.170 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:49.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:49.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:49.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:49.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:49 smithi038 ceph-mon[29779]: pgmap v497: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 652 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-04T16:13:49.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:49 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:49.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:49 smithi038 ceph-mon[38113]: pgmap v497: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 652 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-04T16:13:49.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:49 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:49.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:49.535 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:49 smithi176 ceph-mon[33619]: pgmap v497: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 652 B/s rd, 59 MiB/s wr, 944 op/s 2024-03-04T16:13:49.535 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:49 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:13:49.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 38 16 70208 70192 115.433 75.125 0.00997387 0.00865808 2024-03-04T16:13:49.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:49.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:49.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:49.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:49.903 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:49.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:50.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:50.047 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:50.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:50.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:50.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:50.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:50.303 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:13:50.329 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:50.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.466 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:50.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2004202783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:50 smithi038 ceph-mon[29779]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2004202783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:50 smithi038 ceph-mon[38113]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 39 16 71434 71418 114.438 76.625 0.00389121 0.00873257 2024-03-04T16:13:50.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2004202783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:50 smithi176 ceph-mon[33619]: from='client.25348 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:13:50.610 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:50.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:50.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.749 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:50.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:50.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:50.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:50.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:50.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:51.018 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:51.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:51.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:51.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:51.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:51.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:51.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:51.543 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:51.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:13:51.544830+0000 min lat: 0.00128089 max lat: 0.151627 avg lat: 0.0087974 2024-03-04T16:13:51.545 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:13:51.545 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 40 16 72706 72690 113.564 79.5 0.0100658 0.0087974 2024-03-04T16:13:51.577 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:13:51.577 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: pgmap v498: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: osdmap e262: 8 total, 8 up, 6 in 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.578 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:13:51.579 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:51 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:13:51.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:51.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:51.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.734 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: pgmap v498: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: osdmap e262: 8 total, 8 up, 6 in 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: pgmap v498: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-03-04T16:13:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='client.25348 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: osdmap e262: 8 total, 8 up, 6 in 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:13:51.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:51 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:13:51.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:51.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:51.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:52.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.015 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:52.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:52.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:52.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:52.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:52.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:52.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:52.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:52.438 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:52.540 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:52.544 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 41 15 74119 74104 112.95 88.375 0.00643325 0.00884016 2024-03-04T16:13:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.586 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:52.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:52 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi038 to 3249M 2024-03-04T16:13:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:52 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:13:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:52 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:13:52.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:52 smithi176 ceph-mon[33619]: pgmap v500: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:13:52.687 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:52.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:13:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.725 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:52.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi038 to 3249M 2024-03-04T16:13:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:13:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:13:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[29779]: pgmap v500: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:13:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi038 to 3249M 2024-03-04T16:13:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:13:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:13:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:52 smithi038 ceph-mon[38113]: pgmap v500: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-04T16:13:52.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:52.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:52.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:52.978 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:53.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:53.110 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:53.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:53.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:53.245 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:13:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:13:53.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:53.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:53.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.391 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: Adjusting osd_memory_target on smithi176 to 3248M 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:53.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.393 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:53.393 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:13:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:13:53.393 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:13:53.393 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:13:53.393 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: Adjusting osd_memory_target on smithi176 to 3248M 2024-03-04T16:13:53.394 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.395 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:53.395 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:53.395 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.395 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:53.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:53.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:13:53.451 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:53.537 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:13:53.537 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: Adjusting osd_memory_target on smithi176 to 3248M 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:13:53.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:13:53.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:13:53.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:13:53.545 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 42 16 75432 75416 112.21 82 0.0115495 0.00890724 2024-03-04T16:13:53.553 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.593 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:53.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:53.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:53.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:53.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:53.842 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:13:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:53.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:53.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:53.895 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:13:53.945 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:13:53.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:53.991 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:13:54.020 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:54.023 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:54.035 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:13:54.094 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:13:54.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:54.143 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:13:54.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:54.192 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:13:54.200 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:13:54.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:54.335 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-04T16:13:54.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:54.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:54.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: pgmap v501: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/663887585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/705183850' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:54 smithi176 ceph-mon[33619]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.545 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 43 16 76616 76600 111.322 74 0.00874611 0.00897826 2024-03-04T16:13:54.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:54.623 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: pgmap v501: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/663887585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/705183850' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[29779]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: pgmap v501: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/663887585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/705183850' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:54 smithi038 ceph-mon[38113]: from='client.35369 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:13:54.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:54.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:54.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:54.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:55.038 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:13:55.038 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:13:55.038 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:13:55.038 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:13:55.038 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:13:55.102 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:55.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:55.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:55.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:55.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: osdmap e263: 8 total, 8 up, 6 in 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:13:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:55 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:13:55.394 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.418 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:55.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:55.545 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 44 16 77794 77778 110.465 73.625 0.00895129 0.00904775 2024-03-04T16:13:55.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:55.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: osdmap e263: 8 total, 8 up, 6 in 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:13:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:13:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: from='client.35369 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:13:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: osdmap e263: 8 total, 8 up, 6 in 2024-03-04T16:13:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:13:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:55 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:13:55.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:55.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:55.962 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:55.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:56.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:13:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:56.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:13:56.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:56.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:56.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 5 1 2024-03-04T16:13:56.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:56.382 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:56.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[29779]: pgmap v503: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:13:56.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:13:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:13:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:13:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:13:56.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:56 smithi038 ceph-mon[38113]: pgmap v503: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-04T16:13:56.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:56.548 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 45 16 79002 78986 109.681 75.5 0.01283 0.00911228 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:13:56.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:56 smithi176 ceph-mon[33619]: pgmap v503: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-04T16:13:56.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.668 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:56.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:56.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:56.801 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:56.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:56.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:56.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:57.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:57.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:57.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:57.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:57.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:13:57.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:57.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:57.420 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:57.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:57.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:57.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 46 16 80189 80173 108.908 74.1875 0.00653492 0.00917657 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:13:57.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:13:57.579 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:57 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:13:57.589 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:13:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:13:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:13:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:13:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:13:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:13:57.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:57 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:13:57.621 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:57.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:57.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:57.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:13:57.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:57.929 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:57.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:58.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.079 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:58.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:58.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:58.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:58.262 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:13:58.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:58.368 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:13:58.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:58.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:58.514 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:58.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:58.549 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 47 16 81544 81528 108.392 84.6875 0.0108731 0.00922107 2024-03-04T16:13:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:58 smithi176 ceph-mon[33619]: pgmap v504: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 808 op/s 2024-03-04T16:13:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:58.616 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:13:58.658 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:13:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:58 smithi038 ceph-mon[29779]: pgmap v504: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 808 op/s 2024-03-04T16:13:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:58 smithi038 ceph-mon[38113]: pgmap v504: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 808 op/s 2024-03-04T16:13:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:13:58.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:13:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.795 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:58.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:13:58.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:58.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:58.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:59.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:13:59.036 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:13:59.060 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:13:59.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:59.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:59.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:13:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.342 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:59.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:13:59.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:13:59.485 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:59.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:13:59.549 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 48 16 82925 82909 107.932 86.3125 0.0149881 0.00926054 2024-03-04T16:13:59.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:13:59.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:13:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:13:59.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: osdmap e264: 8 total, 8 up, 6 in 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:13:59.741 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:13:59 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: osdmap e264: 8 total, 8 up, 6 in 2024-03-04T16:13:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1940248712' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: osdmap e264: 8 total, 8 up, 6 in 2024-03-04T16:13:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:13:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:13:59 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:13:59.770 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:59.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:13:59.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:13:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:13:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:13:59.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:00.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:00.057 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:00.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:00.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:00.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:00.341 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:00.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:00.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:00.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:00.549 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 49 16 84216 84200 107.376 80.6875 0.0102513 0.00930796 2024-03-04T16:14:00.561 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:14:00.561 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:14:00.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:14:00.561 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:14:00.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24051 7.7 7:efc91ac2:::benchmark_data_smithi038_77863_object24050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 543.06320952500005, 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.133279604, 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24051 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493320+0000", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493321+0000", 2024-03-04T16:14:00.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3069999999999999e-06 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493378+0000", 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7166000000000001e-05 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493391+0000", 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2524000000000001e-05 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493398+0000", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7810000000000005e-06 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493453+0000", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4936999999999999e-05 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493576+0000", 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122769 2024-03-04T16:14:00.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.493722+0000", 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014567700000000001 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.494623+0000", 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090136199999999995 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497155+0000", 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025317550000000001 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626475+0000", 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12932030999999999 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626541+0000", 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5500999999999995e-05 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626599+0000", 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8514999999999998e-05 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.568 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24060 7.c 7:300854cb:::benchmark_data_smithi038_77863_object24059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 543.06059930699996, 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13071738699999999, 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24060 2024-03-04T16:14:00.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495930+0000", 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.495931+0000", 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2669999999999999e-06 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496021+0000", 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0263e-05 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496036+0000", 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4617000000000001e-05 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496042+0000", 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0730000000000001e-06 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496109+0000", 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6859999999999993e-05 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496221+0000", 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112287 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.496318+0000", 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6948999999999996e-05 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497400+0000", 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010824020000000001 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.497825+0000", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00042419400000000003 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626496+0000", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12867089500000001 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626545+0000", 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9505000000000001e-05 2024-03-04T16:14:00.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.626647+0000", 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102075 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:24067 7.c 7:341347ef:::benchmark_data_smithi038_77863_object24066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 543.05713727700004, 2024-03-04T16:14:00.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.128935775, 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24067 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:14:00.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499392+0000", 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499394+0000", 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.99e-06 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499451+0000", 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6966999999999999e-05 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499523+0000", 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2533000000000005e-05 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499531+0000", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9370000000000004e-06 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499584+0000", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2679999999999997e-05 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499708+0000", 2024-03-04T16:14:00.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012427799999999999 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.499838+0000", 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129534 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.500954+0000", 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001116548 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.580 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.501233+0000", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027919700000000002 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628217+0000", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.126983873 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628271+0000", 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3789000000000003e-05 2024-03-04T16:14:00.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:04:57.628328+0000", 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6449000000000001e-05 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83617 7.d 7:b5cd7487:::benchmark_data_smithi038_77863_object83616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 500.276680645, 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11925000199999999, 2024-03-04T16:14:00.582 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83617 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279848+0000", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279850+0000", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8840000000000001e-06 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279911+0000", 2024-03-04T16:14:00.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1161999999999995e-05 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279927+0000", 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5259000000000001e-05 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.279935+0000", 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5909999999999996e-06 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280005+0000", 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0185000000000006e-05 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280258+0000", 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025285800000000001 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280462+0000", 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020391100000000001 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.286072+0000", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056095620000000002 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.288597+0000", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025255910000000002 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.398999+0000", 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.110401524 2024-03-04T16:14:00.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.399039+0000", 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9724000000000001e-05 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.399098+0000", 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9750999999999998e-05 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:83619 7.d 7:bd3820de:::benchmark_data_smithi038_77863_object83618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 500.275817958, 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.119792248, 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 83619 2024-03-04T16:14:00.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280711+0000", 2024-03-04T16:14:00.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280713+0000", 2024-03-04T16:14:00.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:00.592 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.852e-06 2024-03-04T16:14:00.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280746+0000", 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3065000000000003e-05 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280758+0000", 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1905e-05 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.280769+0000", 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0851e-05 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283138+0000", 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023693049999999999 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283302+0000", 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000164069 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.283457+0000", 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015461000000000001 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.286240+0000", 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027835030000000001 2024-03-04T16:14:00.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.288696+0000", 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024562360000000001 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400382+0000", 2024-03-04T16:14:00.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.111685189 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400445+0000", 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3618e-05 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:40.400503+0000", 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8044999999999997e-05 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.597 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84320 7.c 7:3115064a:::benchmark_data_smithi038_77863_object84319:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 499.36949053000001, 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.152879031, 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84320 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.598 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187039+0000", 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187042+0000", 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.994e-06 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187094+0000", 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2120000000000002e-05 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187111+0000", 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7541999999999999e-05 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187120+0000", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7110000000000001e-06 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187189+0000", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9076000000000001e-05 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187372+0000", 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000183313 2024-03-04T16:14:00.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.187533+0000", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016118799999999999 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.190831+0000", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032979760000000002 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.191323+0000", 2024-03-04T16:14:00.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049143600000000004 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339800+0000", 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14847677000000001 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339865+0000", 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.533e-05 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.339918+0000", 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2574999999999997e-05 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84340 7.7 7:e70ccc3e:::benchmark_data_smithi038_77863_object84339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 499.33834647899999, 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14684087500000001, 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.604 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84340 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218183+0000", 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218185+0000", 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9860000000000001e-06 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218225+0000", 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0855000000000001e-05 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218236+0000", 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.097e-05 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218243+0000", 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9380000000000003e-06 2024-03-04T16:14:00.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218314+0000", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1088999999999996e-05 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218446+0000", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131662 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.218589+0000", 2024-03-04T16:14:00.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000143261 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.220839+0000", 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002249282 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.221181+0000", 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034215299999999999 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.364867+0000", 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.143686181 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.364938+0000", 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0876000000000004e-05 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365023+0000", 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5622000000000001e-05 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.610 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84341 7.c 7:36ea4711:::benchmark_data_smithi038_77863_object84340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 499.32867437499999, 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.121028788, 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84341 2024-03-04T16:14:00.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227855+0000", 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227856+0000", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3340000000000001e-06 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227897+0000", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0918999999999999e-05 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227906+0000", 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4099999999999997e-06 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227912+0000", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5099999999999998e-06 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.227961+0000", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9387e-05 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228096+0000", 2024-03-04T16:14:00.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013440899999999999 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228252+0000", 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015648800000000001 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.230424+0000", 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021721790000000002 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.231177+0000", 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00075235999999999999 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348763+0000", 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.117586044 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348830+0000", 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7508000000000005e-05 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.348883+0000", 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3239999999999998e-05 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84342 7.7 7:ee3774c8:::benchmark_data_smithi038_77863_object84341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 499.32808089399998, 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.136723498, 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84342 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228448+0000", 2024-03-04T16:14:00.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228449+0000", 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.08e-06 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228512+0000", 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2354000000000006e-05 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228521+0000", 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7049999999999994e-06 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228528+0000", 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9439999999999999e-06 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228582+0000", 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3842999999999999e-05 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228698+0000", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116078 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.228827+0000", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012882400000000001 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.231154+0000", 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023271849999999998 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232107+0000", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00095312599999999999 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365079+0000", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13297209800000001 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365133+0000", 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3532999999999997e-05 2024-03-04T16:14:00.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365172+0000", 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8727999999999999e-05 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:84344 7.7 7:ef5cf9b4:::benchmark_data_smithi038_77863_object84343:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 499.324031805, 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.132825691, 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.623 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 84344 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232497+0000", 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232499+0000", 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6390000000000001e-06 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232550+0000", 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1088000000000003e-05 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232559+0000", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3190000000000001e-06 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232565+0000", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2229999999999998e-06 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232635+0000", 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0155999999999997e-05 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232718+0000", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3217e-05 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.232809+0000", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1291000000000004e-05 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.235006+0000", 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021970459999999998 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.236595+0000", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001589177 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365224+0000", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12862828400000001 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365278+0000", 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4613e-05 2024-03-04T16:14:00.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:05:41.365323+0000", 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4638000000000002e-05 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:138242 7.7 7:e7af825f:::benchmark_data_smithi038_77863_object19974:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 443.04781615299999, 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13032482200000001, 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 138242 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508713+0000", 2024-03-04T16:14:00.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508714+0000", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.26e-06 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508720+0000", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3460000000000001e-06 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508729+0000", 2024-03-04T16:14:00.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8689999999999999e-06 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508737+0000", 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9780000000000005e-06 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508801+0000", 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3350999999999997e-05 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.508997+0000", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019587599999999999 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.509104+0000", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107469 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.514111+0000", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050073970000000002 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.514864+0000", 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00075243800000000002 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.638975+0000", 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.124111007 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.639007+0000", 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2280000000000003e-05 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.639038+0000", 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0550999999999999e-05 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.635 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:138252 7.7 7:ef9db293:::benchmark_data_smithi038_77863_object19984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 443.03699305499998, 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12241484399999999, 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.636 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 138252 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519536+0000", 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519538+0000", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.841e-06 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519543+0000", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6400000000000002e-06 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519555+0000", 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1358999999999999e-05 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.638 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519563+0000", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7759999999999997e-06 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519632+0000", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9832000000000005e-05 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519781+0000", 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014851 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.639 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.519910+0000", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012902399999999999 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.523610+0000", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037004020000000002 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.524514+0000", 2024-03-04T16:14:00.640 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090372900000000003 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641877+0000", 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11736239 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641918+0000", 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1087999999999997e-05 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.641 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:06:37.641951+0000", 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3253000000000002e-05 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159707 7.d 7:b4f69bdc:::benchmark_data_smithi038_77863_object41439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 419.91863218899999, 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.140372144, 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.642 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159707 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637897+0000", 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.643 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637899+0000", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.792e-06 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637904+0000", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6570000000000002e-06 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637918+0000", 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4014e-05 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.644 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637927+0000", 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2390000000000002e-06 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.637988+0000", 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1268999999999999e-05 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.645 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638102+0000", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114027 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.638185+0000", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3429000000000004e-05 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.640863+0000", 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026778000000000001 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.646 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.768307+0000", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12744344799999999 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778170+0000", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0098631480000000004 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778221+0000", 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1202000000000002e-05 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.647 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778269+0000", 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8118999999999998e-05 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159712 7.d 7:b9855caa:::benchmark_data_smithi038_77863_object41444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 419.913825288, 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.135708674, 2024-03-04T16:14:00.648 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: 6.19 deep-scrub starts 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: 6.19 deep-scrub ok 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: pgmap v506: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:14:00.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:00 smithi176 ceph-mon[33619]: osdmap e265: 8 total, 8 up, 6 in 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159712 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.650 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642704+0000", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642705+0000", 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3430000000000001e-06 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.651 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642710+0000", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4780000000000002e-06 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642718+0000", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8410000000000006e-06 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642724+0000", 2024-03-04T16:14:00.652 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2669999999999997e-06 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642760+0000", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6625000000000003e-05 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642844+0000", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3880000000000003e-05 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642918+0000", 2024-03-04T16:14:00.653 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3978000000000002e-05 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.647940+0000", 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0050219000000000001 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.768391+0000", 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12045135799999999 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.654 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778324+0000", 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0099328090000000008 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778374+0000", 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0189999999999999e-05 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778412+0000", 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8005e-05 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159713 7.7 7:e219d9b5:::benchmark_data_smithi038_77863_object41445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 419.91379284300001, 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13068476400000001, 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.656 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159713 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642736+0000", 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642737+0000", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0500000000000003e-07 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642740+0000", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7140000000000002e-06 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642745+0000", 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6749999999999999e-06 2024-03-04T16:14:00.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642749+0000", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9600000000000002e-06 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642810+0000", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0588999999999999e-05 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.642920+0000", 2024-03-04T16:14:00.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000109808 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.643036+0000", 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116013 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645568+0000", 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025322019999999999 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.772612+0000", 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12704433300000001 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773350+0000", 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00073744000000000001 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773386+0000", 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6205000000000003e-05 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.661 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.773421+0000", 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5219999999999998e-05 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:159717 7.d 7:b0b31d2c:::benchmark_data_smithi038_77863_object41449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 419.91177375900003, 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.13377419199999999, 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 159717 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644755+0000", 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644757+0000", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.333e-06 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644761+0000", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2570000000000001e-06 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644771+0000", 2024-03-04T16:14:00.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0434999999999999e-05 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.644777+0000", 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6409999999999999e-06 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645495+0000", 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00071810600000000004 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645614+0000", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011852499999999999 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.645732+0000", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118632 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.648042+0000", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002309644 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.776738+0000", 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12869599000000001 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778452+0000", 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017144409999999999 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778496+0000", 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3905000000000001e-05 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:00.778529+0000", 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3283e-05 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163737 7.7 7:e92a9127:::benchmark_data_smithi038_77863_object45469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 415.38038010399998, 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1836797790000002, 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:14:00.668 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163737 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176149+0000", 2024-03-04T16:14:00.669 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176151+0000", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7600000000000001e-06 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176156+0000", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9230000000000001e-06 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176167+0000", 2024-03-04T16:14:00.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1066e-05 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176173+0000", 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.319e-06 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176237+0000", 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3984000000000003e-05 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176404+0000", 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016733899999999999 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.176531+0000", 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012694900000000001 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.179756+0000", 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003224401 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.193019+0000", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013263138000000001 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204566+0000", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011546910000000001 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204579+0000", 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2941e-05 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215396+0000", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.010817297 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215422+0000", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5999999999999998e-05 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.359741+0000", 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.144318645 2024-03-04T16:14:00.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.359829+0000", 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8107000000000002e-05 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163746 7.7 7:ee84827a:::benchmark_data_smithi038_77863_object45478:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 415.37345567099999, 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.177985804, 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.675 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163746 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183073+0000", 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183076+0000", 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3640000000000002e-06 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.678 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.678 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183082+0000", 2024-03-04T16:14:00.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5080000000000002e-06 2024-03-04T16:14:00.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183097+0000", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4764e-05 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183105+0000", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4190000000000002e-06 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183161+0000", 2024-03-04T16:14:00.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5736000000000003e-05 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183296+0000", 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000134661 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.183415+0000", 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118998 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.189322+0000", 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059075170000000001 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204612+0000", 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015290056 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.204626+0000", 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3361e-05 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215543+0000", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0109174489999999 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215555+0000", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1612e-05 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360112+0000", 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1445573200000001 2024-03-04T16:14:00.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.361059+0000", 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000947039 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1414 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93955426370688 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 415.36880743500001, 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.048487942, 2024-03-04T16:14:00.683 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1414 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187722+0000", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187724+0000", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5770000000000001e-06 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187731+0000", 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6370000000000002e-06 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.685 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187750+0000", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9174999999999999e-05 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187761+0000", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1079e-05 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187835+0000", 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3913999999999997e-05 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.686 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op must wait for map", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.187844+0000", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6999999999999997e-06 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.201860+0000", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014015929 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.201870+0000", 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0411000000000001e-05 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.687 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.228681+0000", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.026810666 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.229187+0000", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00050616300000000001 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.233029+0000", 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038424380000000001 2024-03-04T16:14:00.688 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.233132+0000", 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010299299999999999 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.236210+0000", 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030772600000000001 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163754 7.7 7:e3a25a20:::benchmark_data_smithi038_77863_object45486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 415.34448363299998, 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.189298231, 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163754 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:14:00.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212045+0000", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212047+0000", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.007e-06 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:14:00.691 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212054+0000", 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3670000000000002e-06 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212064+0000", 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0253e-05 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212072+0000", 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1310000000000006e-06 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212135+0000", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2322000000000004e-05 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.212146+0000", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1117999999999999e-05 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215584+0000", 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.00343866 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.215593+0000", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0699999999999996e-06 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360201+0000", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1446078790000001 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360324+0000", 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122609 2024-03-04T16:14:00.694 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,3", 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360772+0000", 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044833200000000002 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.371865+0000", 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011092328 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.695 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.383538+0000", 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011673174 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.394054+0000", 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010515871 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401133+0000", 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070789609999999999 2024-03-04T16:14:00.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401200+0000", 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7162000000000005e-05 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.401344+0000", 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014398699999999999 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:14:00.697 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:14:00.698 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:14:00.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: 6.19 deep-scrub starts 2024-03-04T16:14:00.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: 6.19 deep-scrub ok 2024-03-04T16:14:00.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: pgmap v506: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[29779]: osdmap e265: 8 total, 8 up, 6 in 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: 6.19 deep-scrub starts 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: 6.19 deep-scrub ok 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: pgmap v506: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:14:00.699 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:00 smithi038 ceph-mon[38113]: osdmap e265: 8 total, 8 up, 6 in 2024-03-04T16:14:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:00.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:00.801 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:00.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:00.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:00.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:00.984 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:00.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:01.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:01.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:01.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:01.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:01.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:01.382 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-04T16:14:01.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:01.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:01.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:01.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:01.549 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 50 16 85536 85520 106.878 82.5 0.00722286 0.0093514 2024-03-04T16:14:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:01.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:14:01.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:01.652 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:01.652 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:01.653 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:14:01.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:01.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:01.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:14:01.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:01.716 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:14:01.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.828 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:01.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:01.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:14:01.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:01.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:01.968 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:01.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:02.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:02.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.115 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:02.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:02.138 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:14:02.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:02.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:02.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:02.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 7 1 2024-03-04T16:14:02.395 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:02.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:02.452 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:14:02.452 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-mon[33619]: osdmap e266: 8 total, 8 up, 6 in 2024-03-04T16:14:02.452 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:14:02.452 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:14:02.453 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-mon[33619]: pgmap v509: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-04T16:14:02.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:02.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:02.550 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 51 16 86818 86802 106.352 80.125 0.0146325 0.00939805 2024-03-04T16:14:02.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:02.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:02.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:14:02.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[29779]: osdmap e266: 8 total, 8 up, 6 in 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[29779]: pgmap v509: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/616383806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[38113]: osdmap e266: 8 total, 8 up, 6 in 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:14:02.692 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:02 smithi038 ceph-mon[38113]: pgmap v509: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 63 MiB/s wr, 1.02k op/s 2024-03-04T16:14:02.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:02.838 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:02.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:02.971 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:02.986 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:02.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:03.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:03.093 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:14:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:03.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:03.132 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:14:03.169 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:14:03.208 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:14:03.224 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:03.249 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:14:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:03.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:03.300 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:14:03.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:03.342 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:14:03.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:03.385 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:14:03.388 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:14:03.389 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:14:03.403 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:03.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:03.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:14:03.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:14:03.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:03.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/956909875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:03.535 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:03.535 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:14:03.535 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:14:03.535 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:03.535 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:03 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/956909875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:03.543 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:03.550 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 52 16 88127 88111 105.88 81.8125 0.00911892 0.00943993 2024-03-04T16:14:03.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:03 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:14:03.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:03 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:14:03.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:03.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:03 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/956909875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:03.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:03.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:03.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:03.836 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:03.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:03.981 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:03.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:04.083 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:04.113 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:14:04.113 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:14:04.127 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:04.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.264 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:04.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:04.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:04.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:04.533 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:04.536 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:04.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 53 16 89553 89537 105.562 89.125 0.00725615 0.0094682 2024-03-04T16:14:04.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:04 smithi038 ceph-mon[29779]: pgmap v510: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-04T16:14:04.623 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:04 smithi038 ceph-mon[38113]: pgmap v510: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-04T16:14:04.638 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:04.674 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:04.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:04 smithi176 ceph-mon[33619]: pgmap v510: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-04T16:14:04.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:14:04.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:04.811 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:04.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:04.951 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:04.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:05.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:05.091 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:05.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:05.198 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:14:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.240 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:05.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:05.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:05.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:05.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:05.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:14:05.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:14:05.487 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[29779]: osdmap e267: 8 total, 8 up, 6 in 2024-03-04T16:14:05.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:14:05.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:14:05.488 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:05 smithi038 ceph-mon[38113]: osdmap e267: 8 total, 8 up, 6 in 2024-03-04T16:14:05.519 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:05.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:05.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 54 16 90953 90937 105.227 87.5 0.0075344 0.00949891 2024-03-04T16:14:05.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:05 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:14:05.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:05 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:14:05.585 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:05 smithi176 ceph-mon[33619]: osdmap e267: 8 total, 8 up, 6 in 2024-03-04T16:14:05.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:05.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:05.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:05.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:05.899 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:05.931 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:05.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:06.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:06.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:06.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:06.208 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:06.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:06.348 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:06.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:06.454 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:14:06.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:06.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 55 16 92305 92289 104.85 84.5 0.00830807 0.00953084 2024-03-04T16:14:06.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:06.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:06.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: pgmap v512: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:14:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: pgmap v512: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:14:06.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:14:06.767 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:06.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:06.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:06.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:14:06.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:14:06.853 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: pgmap v512: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-04T16:14:06.854 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:14:06.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:06.908 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:06.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:06.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:14:07.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:07.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:07.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.202 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:07.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:07.303 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.337 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:07.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:07.374 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:14:07.374 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:14:07.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:07.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:07.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 56 16 93615 93599 104.44 81.875 0.00743097 0.00957023 2024-03-04T16:14:07.582 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:07.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:07.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:14:07.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[29779]: osdmap e268: 8 total, 8 up, 6 in 2024-03-04T16:14:07.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:14:07.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:14:07.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:14:07.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:14:07.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:07.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:14:07.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:07 smithi038 ceph-mon[38113]: osdmap e268: 8 total, 8 up, 6 in 2024-03-04T16:14:07.765 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1698705381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:14:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:07 smithi176 ceph-mon[33619]: osdmap e268: 8 total, 8 up, 6 in 2024-03-04T16:14:07.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:07.903 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:07.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:08.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:08.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:08.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:08.194 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:08.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:08.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:14:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:08.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:08.436 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:08.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:08.551 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 57 16 94994 94978 104.12 86.1875 0.00637573 0.00959959 2024-03-04T16:14:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:08 smithi038 ceph-mon[29779]: pgmap v514: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 832 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-04T16:14:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:08.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:08 smithi038 ceph-mon[38113]: pgmap v514: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 832 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-04T16:14:08.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:08.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:08 smithi176 ceph-mon[33619]: pgmap v514: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 832 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-04T16:14:08.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:08.645 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:08.657 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:08.714 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:14:08.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:08.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:08.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:08.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:08.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:08.933 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:08.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:09.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:09.065 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:09.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.206 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:09.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:09.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:09.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:09.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:09.478 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-04T16:14:09.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:09.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:14:09.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:14:09.552 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 58 16 96310 96294 103.743 82.25 0.0159568 0.00963481 2024-03-04T16:14:09.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:09.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:09.727 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:09.761 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:14:09.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:09.898 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:09.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:09.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:10.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.044 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:10.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:10.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:14:10.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.191 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:10.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:10.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:10.324 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:10.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:10.470 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:10.553 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 59 16 97704 97688 103.459 87.125 0.0112398 0.00966121 2024-03-04T16:14:10.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:10.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:10.613 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:14:10.613 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:14:10.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[38113]: osdmap e269: 8 total, 8 up, 6 in 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[38113]: pgmap v516: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[29779]: osdmap e269: 8 total, 8 up, 6 in 2024-03-04T16:14:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:10 smithi038 ceph-mon[29779]: pgmap v516: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-04T16:14:10.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:10.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:10 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4000681456' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:14:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:10 smithi176 ceph-mon[33619]: osdmap e269: 8 total, 8 up, 6 in 2024-03-04T16:14:10.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:10 smithi176 ceph-mon[33619]: pgmap v516: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-04T16:14:10.886 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:10.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:10.985 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T16:14:10.985 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:14:10.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:11.051 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:11.080 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:11.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.243 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:11.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:11.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:14:11.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:11.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:11.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:11.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:11.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:14:11.554528+0000 min lat: 0.00128089 max lat: 0.151627 avg lat: 0.00968855 2024-03-04T16:14:11.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:14:11.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 60 16 99080 99064 103.167 86 0.00968566 0.00968855 2024-03-04T16:14:11.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:11.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:11.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:11.797 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:11.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:11.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:11.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:12.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:12.075 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:12.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:12.217 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:12.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:12.343 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:12.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:12.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:12.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:12.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 61 16 100400 100384 102.828 82.5 0.0144813 0.00972004 2024-03-04T16:14:12.583 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:12.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:12.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:12.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:12 smithi176 ceph-mon[33619]: pgmap v517: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 815 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-04T16:14:12.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:12.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:12.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:12.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:12 smithi038 ceph-mon[38113]: pgmap v517: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 815 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-04T16:14:12.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:12 smithi038 ceph-mon[29779]: pgmap v517: 148 pgs: 148 active+clean; 5.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 815 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-04T16:14:13.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.058 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:13.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:13.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:13.195 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:13.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:13.320 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:13.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:13.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:13.458 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:13.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 62 16 101677 101661 102.457 79.8125 0.00985716 0.00975577 2024-03-04T16:14:13.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:13.594 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:13.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:13.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:13.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:13.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:13.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:13.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:13.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:13.987 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:13.987 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:13.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:13.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:14.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:14.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:14.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:14.252 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:14.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:14.394 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:14.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:14.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:14.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:14.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:14.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 63 16 102749 102733 101.894 67 0.0137806 0.00980431 2024-03-04T16:14:14.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:14.662 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:14.764 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:14.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:14.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:14 smithi176 ceph-mon[33619]: pgmap v518: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-04T16:14:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/854003203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:14.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:14 smithi038 ceph-mon[29779]: pgmap v518: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-04T16:14:14.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/854003203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:14.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:14 smithi038 ceph-mon[38113]: pgmap v518: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.50k op/s 2024-03-04T16:14:14.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/854003203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:14.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:15.048 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:15.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:15.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:15.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:15.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.463 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:15.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 64 16 103723 103707 101.253 60.875 0.00798754 0.00987173 2024-03-04T16:14:15.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:15.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:14:15.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:14:15.761 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:14:15.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:15.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:15.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:15.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:15.994 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:15.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:16.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:16.130 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:16.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:16.268 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:16.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:16.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.399 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:16.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:16.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:16.536 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:16.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 65 16 105035 105019 100.957 82 0.00416634 0.009901 2024-03-04T16:14:16.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:16.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:16.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:16.803 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:16.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:16.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:16.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:16 smithi038 ceph-mon[29779]: pgmap v519: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-04T16:14:16.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:16 smithi038 ceph-mon[38113]: pgmap v519: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-04T16:14:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:16 smithi176 ceph-mon[33619]: pgmap v519: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.1 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-03-04T16:14:17.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:17.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:17.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:17.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:17.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:17.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:17.555 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 66 15 106060 106045 100.398 64.125 0.0341143 0.0099552 2024-03-04T16:14:17.583 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:17.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:17.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:17.827 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:17.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:17.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:17.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:18.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:18.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:18.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:18.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:18.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:18.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:18.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:18.554 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 67 16 107460 107444 100.205 87.4375 0.00914892 0.00997518 2024-03-04T16:14:18.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:18 smithi038 ceph-mon[29779]: pgmap v520: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:18 smithi038 ceph-mon[38113]: pgmap v520: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:18 smithi176 ceph-mon[33619]: pgmap v520: 148 pgs: 148 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:19.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:19.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:19.348 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:19.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:19.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:19.556 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 68 16 108876 108860 100.031 88.5 0.00875319 0.00999216 2024-03-04T16:14:19.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:19.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:19.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:19.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:19.778 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:19.880 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:19.921 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:19.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:19.992 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:19.997 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:20.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:20.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:20.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.224 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:20.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:20.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:20.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:20.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:20.509 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:20.555 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 69 15 110319 110304 99.8893 90.25 0.0101635 0.010007 2024-03-04T16:14:20.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:20.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:20.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:20.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:20.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:20.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:20 smithi038 ceph-mon[29779]: pgmap v521: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1018 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-04T16:14:20.878 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:20 smithi038 ceph-mon[38113]: pgmap v521: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1018 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-04T16:14:20.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:20.916 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:20.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:20.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:21.018 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:21.049 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:20 smithi176 ceph-mon[33619]: pgmap v521: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1018 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-04T16:14:21.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:21.198 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:21.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:21.339 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:21.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:21.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:21.555 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 70 16 111626 111610 99.6283 81.625 0.00742988 0.0100333 2024-03-04T16:14:21.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:21.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:21.671 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:14:21.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1967166682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:21.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:21 smithi038 ceph-mon[29779]: from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:21.672 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:21 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1967166682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:21.672 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:21 smithi038 ceph-mon[38113]: from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:21.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:21.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:21.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:21.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:21.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:22.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:22.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:22.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:14:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1967166682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:21 smithi176 ceph-mon[33619]: from='client.35411 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:22.108 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:14:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=cleanup t=2024-03-04T16:14:21.738359888Z level=info msg="Completed cleanup jobs" duration=2.934638ms 2024-03-04T16:14:22.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:22.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:22.289 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:22.323 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:22.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:22.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:22.466 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:22.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:22.556 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 71 16 113071 113055 99.497 90.3125 0.00942034 0.010046 2024-03-04T16:14:22.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:22.598 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:22.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:22.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[29779]: pgmap v522: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[29779]: from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[29779]: osdmap e270: 8 total, 8 up, 6 in 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[38113]: pgmap v522: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:14:22.840 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:22.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[38113]: from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:22.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:22 smithi038 ceph-mon[38113]: osdmap e270: 8 total, 8 up, 6 in 2024-03-04T16:14:22.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:22.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:22.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:22.950 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:22 smithi176 ceph-mon[33619]: pgmap v522: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-04T16:14:22.950 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:22 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:22.950 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:22 smithi176 ceph-mon[33619]: from='client.35411 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:22.951 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:22 smithi176 ceph-mon[33619]: osdmap e270: 8 total, 8 up, 6 in 2024-03-04T16:14:22.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:23.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:23.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:23.178 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:23.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:23.318 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:23.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:23.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:23.453 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:23.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:23.556 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 72 16 114519 114503 99.372 90.5 0.00547074 0.0100589 2024-03-04T16:14:23.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:23.648 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:23.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:23.736 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:23.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:23.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:23.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:23.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:24.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:24.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.158 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:24.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:24.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.296 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:24.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:24.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:24.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:24.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:24.556 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 73 16 115978 115962 99.2597 91.1875 0.0171079 0.0100675 2024-03-04T16:14:24.557 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:24.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:14:24.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:24.930 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:24.964 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:24.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:24 smithi038 ceph-mon[29779]: pgmap v524: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:14:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:24 smithi038 ceph-mon[38113]: pgmap v524: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:14:25.057 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:25.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:24 smithi176 ceph-mon[33619]: pgmap v524: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:14:25.119 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:25.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:25.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:25.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:25.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:25.439 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:25.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:25.559 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 74 16 117186 117170 98.9347 75.5 0.00617368 0.0101028 2024-03-04T16:14:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:25.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:25.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.725 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:25.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15456.0:117479 9.5 9:ab3c58e1:::benchmark_data_smithi038_131582_object117478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:14:25.777416+0000", 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.007175954, 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072164769999999998, 2024-03-04T16:14:25.786 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15456", 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3662031207", 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 117479 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777416+0000", 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777416+0000", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777425+0000", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3129999999999999e-06 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777452+0000", 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6985e-05 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777464+0000", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1831000000000001e-05 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777471+0000", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7589999999999997e-06 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.777692+0000", 2024-03-04T16:14:25.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022049799999999999 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778296+0000", 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060463200000000002 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5,7", 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778450+0000", 2024-03-04T16:14:25.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000153776 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15456.0:117476 9.0 e270/242 9:0f033d91:::benchmark_data_smithi038_131582_object117475:head v 270'7247, mlcod=270'7244)", 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:14:25.778929+0000", 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0056636209999999998, 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058086270000000002, 2024-03-04T16:14:25.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778929+0000", 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778929+0000", 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778930+0000", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.64e-07 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778961+0000", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1744e-05 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778973+0000", 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1182999999999999e-05 2024-03-04T16:14:25.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.778979+0000", 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0510000000000002e-06 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.779282+0000", 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00030362500000000002 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:25.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.779301+0000", 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8382000000000001e-05 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15456.0:117487 9.9 9:9922cf91:::benchmark_data_smithi038_131582_object117486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270)", 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:14:25.783349+0000", 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0012431429999999999, 2024-03-04T16:14:25.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001463844, 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15456", 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3662031207", 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 117487 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783349+0000", 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783349+0000", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783351+0000", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7379999999999999e-06 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783375+0000", 2024-03-04T16:14:25.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3669999999999999e-05 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783387+0000", 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2249000000000001e-05 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783395+0000", 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9820000000000008e-06 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:25.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783460+0000", 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5622000000000003e-05 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783603+0000", 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014236500000000001 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 5,7", 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.783750+0000", 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014690100000000001 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:25.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15456.0:117471 9.8 e270/242 9:1e241e85:::benchmark_data_smithi038_131582_object117470:head v 270'7348, mlcod=270'7346)", 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:14:25.772681+0000", 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.011911512000000001, 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012208910999999999, 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772681+0000", 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772681+0000", 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772682+0000", 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.285e-06 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:25.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772710+0000", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8073999999999999e-05 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772720+0000", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4790000000000002e-06 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.772726+0000", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5640000000000002e-06 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:25.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.774735+0000", 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020082659999999999 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.774756+0000", 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1608999999999999e-05 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.15456.0:117478 9.6 e270/242 9:6184f59b:::benchmark_data_smithi038_131582_object117477:head v 270'7415, mlcod=270'7412)", 2024-03-04T16:14:25.803 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:14:25.776054+0000", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0085378820000000001, 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0089125660000000002, 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776054+0000", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776054+0000", 2024-03-04T16:14:25.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776056+0000", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.17e-06 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776176+0000", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119918 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776185+0000", 2024-03-04T16:14:25.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0389999999999993e-06 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776191+0000", 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4099999999999996e-06 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776241+0000", 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9629999999999997e-05 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:25.806 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:14:25.776262+0000", 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0995000000000001e-05 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 5 2024-03-04T16:14:25.807 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:14:25.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.860 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:25.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:25.961 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:25.997 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:25.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:26.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:26.138 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:26.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:26.285 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:26.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:26.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:26.560 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 75 16 118552 118536 98.7528 85.375 0.00658983 0.0101215 2024-03-04T16:14:26.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.618 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:26.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:26.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:26.755 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:26.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:26.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:26.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:26.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:27.032 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:27.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:27.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:27.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:27.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:27.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:27.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:14:27.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:27.441 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:27 smithi038 ceph-mon[29779]: pgmap v525: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:27.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:27 smithi038 ceph-mon[38113]: pgmap v525: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:27.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:27.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:27.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 76 16 120013 119997 98.6513 91.3125 0.00484664 0.0101324 2024-03-04T16:14:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:27 smithi176 ceph-mon[33619]: pgmap v525: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:27.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:27.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:27.865 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:27.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:27.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:28.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:28.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:28.108 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:14:28.119 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:14:28.133 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:28.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:28.189 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:28.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:28.327 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:28.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.470 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:28.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:28.563 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 77 16 121306 121290 98.4196 80.8125 0.0129154 0.0101563 2024-03-04T16:14:28.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:28.604 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:28.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:28.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:28.839 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:28.870 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:28.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:28.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:29.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:29.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.147 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:29.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:29.218 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":526,"stamp":"2024-03-04T16:14:27.531827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7457937758,"num_objects":113988,"num_object_clones":0,"num_object_copies":341964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113988,"num_whiteouts":0,"num_read":2040,"num_read_kb":1886,"num_write":114691,"num_write_kb":7285021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66744,"ondisk_log_size":66744,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":24639460,"kb_used_data":21768268,"kb_used_omap":9,"kb_used_meta":2870582,"kb_avail":537831452,"statfs":{"total":575970213888,"available":550739406848,"internally_reserved":0,"allocated":22290706432,"data_stored":22287698230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":2939476647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,7,8,10,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":836239360,"num_objects":12760,"num_object_clones":0,"num_object_copies":38280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12760,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12760,"num_write_kb":816640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003221"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":447,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652157+0000","last_change":"2024-03-04T16:13:18.470267+0000","last_active":"2024-03-04T16:14:21.652157+0000","last_peered":"2024-03-04T16:14:21.652157+0000","last_clean":"2024-03-04T16:14:21.652157+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:14:21.652157+0000","last_undegraded":"2024-03-04T16:14:21.652157+0000","last_fullsized":"2024-03-04T16:14:21.652157+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:18.470133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.470133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:48.009404+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":443,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.655108+0000","last_change":"2024-03-04T16:12:19.579690+0000","last_active":"2024-03-04T16:14:21.655108+0000","last_peered":"2024-03-04T16:14:21.655108+0000","last_clean":"2024-03-04T16:14:21.655108+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:14:21.655108+0000","last_undegraded":"2024-03-04T16:14:21.655108+0000","last_fullsized":"2024-03-04T16:14:21.655108+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.579557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.579557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":308,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659168+0000","last_change":"2024-03-04T16:13:53.441131+0000","last_active":"2024-03-04T16:14:21.659168+0000","last_peered":"2024-03-04T16:14:21.659168+0000","last_clean":"2024-03-04T16:14:21.659168+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:14:21.659168+0000","last_undegraded":"2024-03-04T16:14:21.659168+0000","last_fullsized":"2024-03-04T16:14:21.659168+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.441019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.441019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.500140+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":488,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.651840+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:14:21.651840+0000","last_peered":"2024-03-04T16:14:21.651840+0000","last_clean":"2024-03-04T16:14:21.651840+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:14:21.651840+0000","last_undegraded":"2024-03-04T16:14:21.651840+0000","last_fullsized":"2024-03-04T16:14:21.651840+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414557+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":437,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664460+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:14:21.664460+0000","last_peered":"2024-03-04T16:14:21.664460+0000","last_clean":"2024-03-04T16:14:21.664460+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:14:21.664460+0000","last_undegraded":"2024-03-04T16:14:21.664460+0000","last_fullsized":"2024-03-04T16:14:21.664460+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747046+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":449,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.651884+0000","last_change":"2024-03-04T16:12:14.551847+0000","last_active":"2024-03-04T16:14:21.651884+0000","last_peered":"2024-03-04T16:14:21.651884+0000","last_clean":"2024-03-04T16:14:21.651884+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:14:21.651884+0000","last_undegraded":"2024-03-04T16:14:21.651884+0000","last_fullsized":"2024-03-04T16:14:21.651884+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.551417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.551417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":456,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663247+0000","last_change":"2024-03-04T16:13:10.472225+0000","last_active":"2024-03-04T16:14:21.663247+0000","last_peered":"2024-03-04T16:14:21.663247+0000","last_clean":"2024-03-04T16:14:21.663247+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:14:21.663247+0000","last_undegraded":"2024-03-04T16:14:21.663247+0000","last_fullsized":"2024-03-04T16:14:21.663247+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:10.471872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:13:10.471872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:35.779419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":556,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664687+0000","last_change":"2024-03-04T16:12:22.299425+0000","last_active":"2024-03-04T16:14:21.664687+0000","last_peered":"2024-03-04T16:14:21.664687+0000","last_clean":"2024-03-04T16:14:21.664687+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:14:21.664687+0000","last_undegraded":"2024-03-04T16:14:21.664687+0000","last_fullsized":"2024-03-04T16:14:21.664687+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:22.299279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.299279+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":295,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661136+0000","last_change":"2024-03-04T16:12:22.616322+0000","last_active":"2024-03-04T16:14:21.661136+0000","last_peered":"2024-03-04T16:14:21.661136+0000","last_clean":"2024-03-04T16:14:21.661136+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:14:21.661136+0000","last_undegraded":"2024-03-04T16:14:21.661136+0000","last_fullsized":"2024-03-04T16:14:21.661136+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:22.616197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.616197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.411764+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":442,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663255+0000","last_change":"2024-03-04T16:12:13.585218+0000","last_active":"2024-03-04T16:14:21.663255+0000","last_peered":"2024-03-04T16:14:21.663255+0000","last_clean":"2024-03-04T16:14:21.663255+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:14:21.663255+0000","last_undegraded":"2024-03-04T16:14:21.663255+0000","last_fullsized":"2024-03-04T16:14:21.663255+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:13.585073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:12:13.585073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":464,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664403+0000","last_change":"2024-03-04T16:12:02.625596+0000","last_active":"2024-03-04T16:14:21.664403+0000","last_peered":"2024-03-04T16:14:21.664403+0000","last_clean":"2024-03-04T16:14:21.664403+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:14:21.664403+0000","last_undegraded":"2024-03-04T16:14:21.664403+0000","last_fullsized":"2024-03-04T16:14:21.664403+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.625486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.625486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":508,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652384+0000","last_change":"2024-03-04T16:11:58.878158+0000","last_active":"2024-03-04T16:14:21.652384+0000","last_peered":"2024-03-04T16:14:21.652384+0000","last_clean":"2024-03-04T16:14:21.652384+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:14:21.652384+0000","last_undegraded":"2024-03-04T16:14:21.652384+0000","last_fullsized":"2024-03-04T16:14:21.652384+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:11:58.878047+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:11:58.878047+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414762+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":454,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662817+0000","last_change":"2024-03-04T16:13:59.118849+0000","last_active":"2024-03-04T16:14:21.662817+0000","last_peered":"2024-03-04T16:14:21.662817+0000","last_clean":"2024-03-04T16:14:21.662817+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:14:21.662817+0000","last_undegraded":"2024-03-04T16:14:21.662817+0000","last_fullsized":"2024-03-04T16:14:21.662817+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.118720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.495900+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":451,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660094+0000","last_change":"2024-03-04T16:12:07.816272+0000","last_active":"2024-03-04T16:14:21.660094+0000","last_peered":"2024-03-04T16:14:21.660094+0000","last_clean":"2024-03-04T16:14:21.660094+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:14:21.660094+0000","last_undegraded":"2024-03-04T16:14:21.660094+0000","last_fullsized":"2024-03-04T16:14:21.660094+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:07.816157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:07.816157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924129+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662422+0000","last_change":"2024-03-04T16:13:56.116452+0000","last_active":"2024-03-04T16:14:21.662422+0000","last_peered":"2024-03-04T16:14:21.662422+0000","last_clean":"2024-03-04T16:14:21.662422+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:14:21.662422+0000","last_undegraded":"2024-03-04T16:14:21.662422+0000","last_fullsized":"2024-03-04T16:14:21.662422+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.116341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.116341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:58.120504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":486,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652837+0000","last_change":"2024-03-04T16:12:15.539669+0000","last_active":"2024-03-04T16:14:21.652837+0000","last_peered":"2024-03-04T16:14:21.652837+0000","last_clean":"2024-03-04T16:14:21.652837+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:14:21.652837+0000","last_undegraded":"2024-03-04T16:14:21.652837+0000","last_fullsized":"2024-03-04T16:14:21.652837+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.539528+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414795+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":446,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652978+0000","last_change":"2024-03-04T16:12:20.525886+0000","last_active":"2024-03-04T16:14:21.652978+0000","last_peered":"2024-03-04T16:14:21.652978+0000","last_clean":"2024-03-04T16:14:21.652978+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:14:21.652978+0000","last_undegraded":"2024-03-04T16:14:21.652978+0000","last_fullsized":"2024-03-04T16:14:21.652978+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:20.525761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.525761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414852+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":451,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658023+0000","last_change":"2024-03-04T16:13:03.552450+0000","last_active":"2024-03-04T16:14:21.658023+0000","last_peered":"2024-03-04T16:14:21.658023+0000","last_clean":"2024-03-04T16:14:21.658023+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:14:21.658023+0000","last_undegraded":"2024-03-04T16:14:21.658023+0000","last_fullsized":"2024-03-04T16:14:21.658023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:03.552289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:03.552289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.718768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":287,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658762+0000","last_change":"2024-03-04T16:12:13.449849+0000","last_active":"2024-03-04T16:14:21.658762+0000","last_peered":"2024-03-04T16:14:21.658762+0000","last_clean":"2024-03-04T16:14:21.658762+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:14:21.658762+0000","last_undegraded":"2024-03-04T16:14:21.658762+0000","last_fullsized":"2024-03-04T16:14:21.658762+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:13.449771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:13.449771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.411793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'28","reported_seq":258,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658514+0000","last_change":"2024-03-04T16:13:18.622039+0000","last_active":"2024-03-04T16:14:21.658514+0000","last_peered":"2024-03-04T16:14:21.658514+0000","last_clean":"2024-03-04T16:14:21.658514+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:14:21.658514+0000","last_undegraded":"2024-03-04T16:14:21.658514+0000","last_fullsized":"2024-03-04T16:14:21.658514+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-03-04T16:13:18.621837+0000","last_deep_scrub":"250'28","last_deep_scrub_stamp":"2024-03-04T16:13:18.621837+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.621837+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:28.207669+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":438,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.665045+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:14:21.665045+0000","last_peered":"2024-03-04T16:14:21.665045+0000","last_clean":"2024-03-04T16:14:21.665045+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:14:21.665045+0000","last_undegraded":"2024-03-04T16:14:21.665045+0000","last_fullsized":"2024-03-04T16:14:21.665045+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747546+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652782+0000","last_change":"2024-03-04T16:13:58.335444+0000","last_active":"2024-03-04T16:14:21.652782+0000","last_peered":"2024-03-04T16:14:21.652782+0000","last_clean":"2024-03-04T16:14:21.652782+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:14:21.652782+0000","last_undegraded":"2024-03-04T16:14:21.652782+0000","last_fullsized":"2024-03-04T16:14:21.652782+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.334939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.334939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.215128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652491+0000","last_change":"2024-03-04T16:13:17.426445+0000","last_active":"2024-03-04T16:14:21.652491+0000","last_peered":"2024-03-04T16:14:21.652491+0000","last_clean":"2024-03-04T16:14:21.652491+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:14:21.652491+0000","last_undegraded":"2024-03-04T16:14:21.652491+0000","last_fullsized":"2024-03-04T16:14:21.652491+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.426276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.426276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:40.807189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":479,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663274+0000","last_change":"2024-03-04T16:13:05.468168+0000","last_active":"2024-03-04T16:14:21.663274+0000","last_peered":"2024-03-04T16:14:21.663274+0000","last_clean":"2024-03-04T16:14:21.663274+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:14:21.663274+0000","last_undegraded":"2024-03-04T16:14:21.663274+0000","last_fullsized":"2024-03-04T16:14:21.663274+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:13:05.468069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.468069+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.807732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":459,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658979+0000","last_change":"2024-03-04T16:13:55.908132+0000","last_active":"2024-03-04T16:14:21.658979+0000","last_peered":"2024-03-04T16:14:21.658979+0000","last_clean":"2024-03-04T16:14:21.658979+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:14:21.658979+0000","last_undegraded":"2024-03-04T16:14:21.658979+0000","last_fullsized":"2024-03-04T16:14:21.658979+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:13:55.907870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.907870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.802762+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":409,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.653425+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:14:21.653425+0000","last_peered":"2024-03-04T16:14:21.653425+0000","last_clean":"2024-03-04T16:14:21.653425+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:14:21.653425+0000","last_undegraded":"2024-03-04T16:14:21.653425+0000","last_fullsized":"2024-03-04T16:14:21.653425+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.655307+0000","last_change":"2024-03-04T16:12:12.526320+0000","last_active":"2024-03-04T16:14:21.655307+0000","last_peered":"2024-03-04T16:14:21.655307+0000","last_clean":"2024-03-04T16:14:21.655307+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:14:21.655307+0000","last_undegraded":"2024-03-04T16:14:21.655307+0000","last_fullsized":"2024-03-04T16:14:21.655307+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:12:12.526200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:12.526200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716391+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":323,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659945+0000","last_change":"2024-03-04T16:13:10.705990+0000","last_active":"2024-03-04T16:14:21.659945+0000","last_peered":"2024-03-04T16:14:21.659945+0000","last_clean":"2024-03-04T16:14:21.659945+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:14:21.659945+0000","last_undegraded":"2024-03-04T16:14:21.659945+0000","last_fullsized":"2024-03-04T16:14:21.659945+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:13:10.705885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:10.705885+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.412193+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":439,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658168+0000","last_change":"2024-03-04T16:12:23.777697+0000","last_active":"2024-03-04T16:14:21.658168+0000","last_peered":"2024-03-04T16:14:21.658168+0000","last_clean":"2024-03-04T16:14:21.658168+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:14:21.658168+0000","last_undegraded":"2024-03-04T16:14:21.658168+0000","last_fullsized":"2024-03-04T16:14:21.658168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.777579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.777579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716361+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":84,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663107+0000","last_change":"2024-03-04T16:12:23.623101+0000","last_active":"2024-03-04T16:14:21.663107+0000","last_peered":"2024-03-04T16:14:21.663107+0000","last_clean":"2024-03-04T16:14:21.663107+0000","last_became_active":"2024-03-04T16:12:23.622602+0000","last_became_peered":"2024-03-04T16:12:23.622602+0000","last_unstale":"2024-03-04T16:14:21.663107+0000","last_undegraded":"2024-03-04T16:14:21.663107+0000","last_fullsized":"2024-03-04T16:14:21.663107+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.487604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.487604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747659+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":463,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661212+0000","last_change":"2024-03-04T16:13:19.603451+0000","last_active":"2024-03-04T16:14:21.661212+0000","last_peered":"2024-03-04T16:14:21.661212+0000","last_clean":"2024-03-04T16:14:21.661212+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:14:21.661212+0000","last_undegraded":"2024-03-04T16:14:21.661212+0000","last_fullsized":"2024-03-04T16:14:21.661212+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.603191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.603191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:36.865463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":233,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664453+0000","last_change":"2024-03-04T16:12:22.748187+0000","last_active":"2024-03-04T16:14:21.664453+0000","last_peered":"2024-03-04T16:14:21.664453+0000","last_clean":"2024-03-04T16:14:21.664453+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:14:21.664453+0000","last_undegraded":"2024-03-04T16:14:21.664453+0000","last_fullsized":"2024-03-04T16:14:21.664453+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:12:22.748049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.748049+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":453,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660160+0000","last_change":"2024-03-04T16:12:01.877823+0000","last_active":"2024-03-04T16:14:21.660160+0000","last_peered":"2024-03-04T16:14:21.660160+0000","last_clean":"2024-03-04T16:14:21.660160+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:14:21.660160+0000","last_undegraded":"2024-03-04T16:14:21.660160+0000","last_fullsized":"2024-03-04T16:14:21.660160+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:01.877704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:12:01.877704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":171,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.651946+0000","last_change":"2024-03-04T16:13:12.406817+0000","last_active":"2024-03-04T16:14:21.651946+0000","last_peered":"2024-03-04T16:14:21.651946+0000","last_clean":"2024-03-04T16:14:21.651946+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:14:21.651946+0000","last_undegraded":"2024-03-04T16:14:21.651946+0000","last_fullsized":"2024-03-04T16:14:21.651946+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.406690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.406690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.498273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":295,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659073+0000","last_change":"2024-03-04T16:14:05.418615+0000","last_active":"2024-03-04T16:14:21.659073+0000","last_peered":"2024-03-04T16:14:21.659073+0000","last_clean":"2024-03-04T16:14:21.659073+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:14:21.659073+0000","last_undegraded":"2024-03-04T16:14:21.659073+0000","last_fullsized":"2024-03-04T16:14:21.659073+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:05.418476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:05.418476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.527281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":464,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661406+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:14:21.661406+0000","last_peered":"2024-03-04T16:14:21.661406+0000","last_clean":"2024-03-04T16:14:21.661406+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:14:21.661406+0000","last_undegraded":"2024-03-04T16:14:21.661406+0000","last_fullsized":"2024-03-04T16:14:21.661406+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":449,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664655+0000","last_change":"2024-03-04T16:12:14.287844+0000","last_active":"2024-03-04T16:14:21.664655+0000","last_peered":"2024-03-04T16:14:21.664655+0000","last_clean":"2024-03-04T16:14:21.664655+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:14:21.664655+0000","last_undegraded":"2024-03-04T16:14:21.664655+0000","last_fullsized":"2024-03-04T16:14:21.664655+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.287598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747685+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658481+0000","last_change":"2024-03-04T16:13:06.136903+0000","last_active":"2024-03-04T16:14:21.658481+0000","last_peered":"2024-03-04T16:14:21.658481+0000","last_clean":"2024-03-04T16:14:21.658481+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:14:21.658481+0000","last_undegraded":"2024-03-04T16:14:21.658481+0000","last_fullsized":"2024-03-04T16:14:21.658481+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.136769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.136769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:26.987182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663001+0000","last_change":"2024-03-04T16:12:15.574564+0000","last_active":"2024-03-04T16:14:21.663001+0000","last_peered":"2024-03-04T16:14:21.663001+0000","last_clean":"2024-03-04T16:14:21.663001+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:14:21.663001+0000","last_undegraded":"2024-03-04T16:14:21.663001+0000","last_fullsized":"2024-03-04T16:14:21.663001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:15.574491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.574491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":488,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664575+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:14:21.664575+0000","last_peered":"2024-03-04T16:14:21.664575+0000","last_clean":"2024-03-04T16:14:21.664575+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:14:21.664575+0000","last_undegraded":"2024-03-04T16:14:21.664575+0000","last_fullsized":"2024-03-04T16:14:21.664575+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":287,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661220+0000","last_change":"2024-03-04T16:12:19.525429+0000","last_active":"2024-03-04T16:14:21.661220+0000","last_peered":"2024-03-04T16:14:21.661220+0000","last_clean":"2024-03-04T16:14:21.661220+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:14:21.661220+0000","last_undegraded":"2024-03-04T16:14:21.661220+0000","last_fullsized":"2024-03-04T16:14:21.661220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.525317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.525317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.411849+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":453,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664343+0000","last_change":"2024-03-04T16:13:04.431970+0000","last_active":"2024-03-04T16:14:21.664343+0000","last_peered":"2024-03-04T16:14:21.664343+0000","last_clean":"2024-03-04T16:14:21.664343+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:14:21.664343+0000","last_undegraded":"2024-03-04T16:14:21.664343+0000","last_fullsized":"2024-03-04T16:14:21.664343+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.431851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.431851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.749773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":459,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660740+0000","last_change":"2024-03-04T16:13:05.128608+0000","last_active":"2024-03-04T16:14:21.660740+0000","last_peered":"2024-03-04T16:14:21.660740+0000","last_clean":"2024-03-04T16:14:21.660740+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:14:21.660740+0000","last_undegraded":"2024-03-04T16:14:21.660740+0000","last_fullsized":"2024-03-04T16:14:21.660740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.128508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.128508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":516,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661536+0000","last_change":"2024-03-04T16:13:14.622077+0000","last_active":"2024-03-04T16:14:21.661536+0000","last_peered":"2024-03-04T16:14:21.661536+0000","last_clean":"2024-03-04T16:14:21.661536+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:14:21.661536+0000","last_undegraded":"2024-03-04T16:14:21.661536+0000","last_fullsized":"2024-03-04T16:14:21.661536+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:14.621919+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.621919+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.677355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":443,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662851+0000","last_change":"2024-03-04T16:13:13.530157+0000","last_active":"2024-03-04T16:14:21.662851+0000","last_peered":"2024-03-04T16:14:21.662851+0000","last_clean":"2024-03-04T16:14:21.662851+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:14:21.662851+0000","last_undegraded":"2024-03-04T16:14:21.662851+0000","last_fullsized":"2024-03-04T16:14:21.662851+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.530037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.807763+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":451,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661339+0000","last_change":"2024-03-04T16:13:12.612940+0000","last_active":"2024-03-04T16:14:21.661339+0000","last_peered":"2024-03-04T16:14:21.661339+0000","last_clean":"2024-03-04T16:14:21.661339+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:14:21.661339+0000","last_undegraded":"2024-03-04T16:14:21.661339+0000","last_fullsized":"2024-03-04T16:14:21.661339+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.612806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.612806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:37.607460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":466,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652551+0000","last_change":"2024-03-04T16:13:51.392879+0000","last_active":"2024-03-04T16:14:21.652551+0000","last_peered":"2024-03-04T16:14:21.652551+0000","last_clean":"2024-03-04T16:14:21.652551+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:14:21.652551+0000","last_undegraded":"2024-03-04T16:14:21.652551+0000","last_fullsized":"2024-03-04T16:14:21.652551+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:51.392701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:51.392701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.094055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":505,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652735+0000","last_change":"2024-03-04T16:13:01.422169+0000","last_active":"2024-03-04T16:14:21.652735+0000","last_peered":"2024-03-04T16:14:21.652735+0000","last_clean":"2024-03-04T16:14:21.652735+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:14:21.652735+0000","last_undegraded":"2024-03-04T16:14:21.652735+0000","last_fullsized":"2024-03-04T16:14:21.652735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:01.422054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:01.422054+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:31.421973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":321,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659189+0000","last_change":"2024-03-04T16:12:03.904682+0000","last_active":"2024-03-04T16:14:21.659189+0000","last_peered":"2024-03-04T16:14:21.659189+0000","last_clean":"2024-03-04T16:14:21.659189+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:14:21.659189+0000","last_undegraded":"2024-03-04T16:14:21.659189+0000","last_fullsized":"2024-03-04T16:14:21.659189+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:03.903765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.903765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924358+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":612,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664410+0000","last_change":"2024-03-04T16:12:06.530577+0000","last_active":"2024-03-04T16:14:21.664410+0000","last_peered":"2024-03-04T16:14:21.664410+0000","last_clean":"2024-03-04T16:14:21.664410+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:14:21.664410+0000","last_undegraded":"2024-03-04T16:14:21.664410+0000","last_fullsized":"2024-03-04T16:14:21.664410+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:12:06.530482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:06.530482+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":493,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658733+0000","last_change":"2024-03-04T16:13:54.812148+0000","last_active":"2024-03-04T16:14:21.658733+0000","last_peered":"2024-03-04T16:14:21.658733+0000","last_clean":"2024-03-04T16:14:21.658733+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:14:21.658733+0000","last_undegraded":"2024-03-04T16:14:21.658733+0000","last_fullsized":"2024-03-04T16:14:21.658733+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:54.812003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.812003+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:21.284093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":505,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.667821+0000","last_change":"2024-03-04T16:14:03.533418+0000","last_active":"2024-03-04T16:14:21.667821+0000","last_peered":"2024-03-04T16:14:21.667821+0000","last_clean":"2024-03-04T16:14:21.667821+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:14:21.667821+0000","last_undegraded":"2024-03-04T16:14:21.667821+0000","last_fullsized":"2024-03-04T16:14:21.667821+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:14:03.533128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:03.533128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:27.853629+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":445,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.653300+0000","last_change":"2024-03-04T16:12:04.823911+0000","last_active":"2024-03-04T16:14:21.653300+0000","last_peered":"2024-03-04T16:14:21.653300+0000","last_clean":"2024-03-04T16:14:21.653300+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:14:21.653300+0000","last_undegraded":"2024-03-04T16:14:21.653300+0000","last_fullsized":"2024-03-04T16:14:21.653300+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.823773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.823773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414824+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":594,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.668119+0000","last_change":"2024-03-04T16:12:20.287137+0000","last_active":"2024-03-04T16:14:21.668119+0000","last_peered":"2024-03-04T16:14:21.668119+0000","last_clean":"2024-03-04T16:14:21.668119+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:14:21.668119+0000","last_undegraded":"2024-03-04T16:14:21.668119+0000","last_fullsized":"2024-03-04T16:14:21.668119+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:12:20.286793+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.286793+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747715+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":353,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659938+0000","last_change":"2024-03-04T16:13:14.023544+0000","last_active":"2024-03-04T16:14:21.659938+0000","last_peered":"2024-03-04T16:14:21.659938+0000","last_clean":"2024-03-04T16:14:21.659938+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:14:21.659938+0000","last_undegraded":"2024-03-04T16:14:21.659938+0000","last_fullsized":"2024-03-04T16:14:21.659938+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:14.023420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.023420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:33.277251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":479,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664597+0000","last_change":"2024-03-04T16:13:05.639843+0000","last_active":"2024-03-04T16:14:21.664597+0000","last_peered":"2024-03-04T16:14:21.664597+0000","last_clean":"2024-03-04T16:14:21.664597+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:14:21.664597+0000","last_undegraded":"2024-03-04T16:14:21.664597+0000","last_fullsized":"2024-03-04T16:14:21.664597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:05.639731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.639731+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.931458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":171,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652868+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:14:21.652868+0000","last_peered":"2024-03-04T16:14:21.652868+0000","last_clean":"2024-03-04T16:14:21.652868+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:14:21.652868+0000","last_undegraded":"2024-03-04T16:14:21.652868+0000","last_fullsized":"2024-03-04T16:14:21.652868+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":435,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664940+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:14:21.664940+0000","last_peered":"2024-03-04T16:14:21.664940+0000","last_clean":"2024-03-04T16:14:21.664940+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:14:21.664940+0000","last_undegraded":"2024-03-04T16:14:21.664940+0000","last_fullsized":"2024-03-04T16:14:21.664940+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747435+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659654+0000","last_change":"2024-03-04T16:12:00.921309+0000","last_active":"2024-03-04T16:14:21.659654+0000","last_peered":"2024-03-04T16:14:21.659654+0000","last_clean":"2024-03-04T16:14:21.659654+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:14:21.659654+0000","last_undegraded":"2024-03-04T16:14:21.659654+0000","last_fullsized":"2024-03-04T16:14:21.659654+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.921174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.921174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924265+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":314,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660163+0000","last_change":"2024-03-04T16:13:09.675634+0000","last_active":"2024-03-04T16:14:21.660163+0000","last_peered":"2024-03-04T16:14:21.660163+0000","last_clean":"2024-03-04T16:14:21.660163+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:14:21.660163+0000","last_undegraded":"2024-03-04T16:14:21.660163+0000","last_fullsized":"2024-03-04T16:14:21.660163+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:09.675534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:09.675534+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.304692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":453,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660645+0000","last_change":"2024-03-04T16:12:18.923785+0000","last_active":"2024-03-04T16:14:21.660645+0000","last_peered":"2024-03-04T16:14:21.660645+0000","last_clean":"2024-03-04T16:14:21.660645+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:14:21.660645+0000","last_undegraded":"2024-03-04T16:14:21.660645+0000","last_fullsized":"2024-03-04T16:14:21.660645+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:18.923671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.923671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924301+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":450,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662922+0000","last_change":"2024-03-04T16:13:06.668331+0000","last_active":"2024-03-04T16:14:21.662922+0000","last_peered":"2024-03-04T16:14:21.662922+0000","last_clean":"2024-03-04T16:14:21.662922+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:14:21.662922+0000","last_undegraded":"2024-03-04T16:14:21.662922+0000","last_fullsized":"2024-03-04T16:14:21.662922+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.668128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.668128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:31.075704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":357,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661393+0000","last_change":"2024-03-04T16:13:03.675180+0000","last_active":"2024-03-04T16:14:21.661393+0000","last_peered":"2024-03-04T16:14:21.661393+0000","last_clean":"2024-03-04T16:14:21.661393+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:14:21.661393+0000","last_undegraded":"2024-03-04T16:14:21.661393+0000","last_fullsized":"2024-03-04T16:14:21.661393+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:03.675052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:13:03.675052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.412161+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.665142+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:14:21.665142+0000","last_peered":"2024-03-04T16:14:21.665142+0000","last_clean":"2024-03-04T16:14:21.665142+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:14:21.665142+0000","last_undegraded":"2024-03-04T16:14:21.665142+0000","last_fullsized":"2024-03-04T16:14:21.665142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"270'7152","reported_seq":7218,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.241133+0000","last_change":"2024-03-04T16:12:58.261525+0000","last_active":"2024-03-04T16:14:23.241133+0000","last_peered":"2024-03-04T16:14:23.241133+0000","last_clean":"2024-03-04T16:14:23.241133+0000","last_became_active":"2024-03-04T16:12:58.261143+0000","last_became_peered":"2024-03-04T16:12:58.261143+0000","last_unstale":"2024-03-04T16:14:23.241133+0000","last_undegraded":"2024-03-04T16:14:23.241133+0000","last_fullsized":"2024-03-04T16:14:23.241133+0000","mapping_epoch":242,"log_start":"255'3000","ondisk_log_start":"255'3000","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4152,"log_dups_size":0,"ondisk_log_size":4152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.997544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468713472,"num_objects":7152,"num_object_clones":0,"num_object_copies":21456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7152,"num_write_kb":457728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":453,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664171+0000","last_change":"2024-03-04T16:13:04.643016+0000","last_active":"2024-03-04T16:14:21.664171+0000","last_peered":"2024-03-04T16:14:21.664171+0000","last_clean":"2024-03-04T16:14:21.664171+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:14:21.664171+0000","last_undegraded":"2024-03-04T16:14:21.664171+0000","last_fullsized":"2024-03-04T16:14:21.664171+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.642875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.642875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:26.694261+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":496,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659718+0000","last_change":"2024-03-04T16:12:26.939744+0000","last_active":"2024-03-04T16:14:21.659718+0000","last_peered":"2024-03-04T16:14:21.659718+0000","last_clean":"2024-03-04T16:14:21.659718+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:14:21.659718+0000","last_undegraded":"2024-03-04T16:14:21.659718+0000","last_fullsized":"2024-03-04T16:14:21.659718+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:12:26.939557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.939557+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":520,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662025+0000","last_change":"2024-03-04T16:12:03.608848+0000","last_active":"2024-03-04T16:14:21.662025+0000","last_peered":"2024-03-04T16:14:21.662025+0000","last_clean":"2024-03-04T16:14:21.662025+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:14:21.662025+0000","last_undegraded":"2024-03-04T16:14:21.662025+0000","last_fullsized":"2024-03-04T16:14:21.662025+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:03.608639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.608639+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":610,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660067+0000","last_change":"2024-03-04T16:13:02.118333+0000","last_active":"2024-03-04T16:14:21.660067+0000","last_peered":"2024-03-04T16:14:21.660067+0000","last_clean":"2024-03-04T16:14:21.660067+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:14:21.660067+0000","last_undegraded":"2024-03-04T16:14:21.660067+0000","last_fullsized":"2024-03-04T16:14:21.660067+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:13:02.118207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:02.118207+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"270'7170","reported_seq":7235,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.187910+0000","last_change":"2024-03-04T16:12:58.258448+0000","last_active":"2024-03-04T16:14:23.187910+0000","last_peered":"2024-03-04T16:14:23.187910+0000","last_clean":"2024-03-04T16:14:23.187910+0000","last_became_active":"2024-03-04T16:12:58.258106+0000","last_became_peered":"2024-03-04T16:12:58.258106+0000","last_unstale":"2024-03-04T16:14:23.187910+0000","last_undegraded":"2024-03-04T16:14:23.187910+0000","last_fullsized":"2024-03-04T16:14:23.187910+0000","mapping_epoch":242,"log_start":"259'3600","ondisk_log_start":"259'3600","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3570,"log_dups_size":0,"ondisk_log_size":3570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.718735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469893120,"num_objects":7170,"num_object_clones":0,"num_object_copies":21510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7170,"num_write_kb":458880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":440,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664559+0000","last_change":"2024-03-04T16:13:52.548216+0000","last_active":"2024-03-04T16:14:21.664559+0000","last_peered":"2024-03-04T16:14:21.664559+0000","last_clean":"2024-03-04T16:14:21.664559+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:14:21.664559+0000","last_undegraded":"2024-03-04T16:14:21.664559+0000","last_fullsized":"2024-03-04T16:14:21.664559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.053562+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659327+0000","last_change":"2024-03-04T16:12:04.845679+0000","last_active":"2024-03-04T16:14:21.659327+0000","last_peered":"2024-03-04T16:14:21.659327+0000","last_clean":"2024-03-04T16:14:21.659327+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:14:21.659327+0000","last_undegraded":"2024-03-04T16:14:21.659327+0000","last_fullsized":"2024-03-04T16:14:21.659327+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.845559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.845559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924154+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":488,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652480+0000","last_change":"2024-03-04T16:13:19.438912+0000","last_active":"2024-03-04T16:14:21.652480+0000","last_peered":"2024-03-04T16:14:21.652480+0000","last_clean":"2024-03-04T16:14:21.652480+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:14:21.652480+0000","last_undegraded":"2024-03-04T16:14:21.652480+0000","last_fullsized":"2024-03-04T16:14:21.652480+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:13:19.438774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.438774+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:30.125115+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":357,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661052+0000","last_change":"2024-03-04T16:13:55.457911+0000","last_active":"2024-03-04T16:14:21.661052+0000","last_peered":"2024-03-04T16:14:21.661052+0000","last_clean":"2024-03-04T16:14:21.661052+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:14:21.661052+0000","last_undegraded":"2024-03-04T16:14:21.661052+0000","last_fullsized":"2024-03-04T16:14:21.661052+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.457585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.457585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:05.916401+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"270'7014","reported_seq":7080,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.636640+0000","last_change":"2024-03-04T16:12:58.259241+0000","last_active":"2024-03-04T16:14:22.636640+0000","last_peered":"2024-03-04T16:14:22.636640+0000","last_clean":"2024-03-04T16:14:22.636640+0000","last_became_active":"2024-03-04T16:12:58.258565+0000","last_became_peered":"2024-03-04T16:12:58.258565+0000","last_unstale":"2024-03-04T16:14:22.636640+0000","last_undegraded":"2024-03-04T16:14:22.636640+0000","last_fullsized":"2024-03-04T16:14:22.636640+0000","mapping_epoch":242,"log_start":"254'2600","ondisk_log_start":"254'2600","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4414,"log_dups_size":0,"ondisk_log_size":4414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:26.372536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459669504,"num_objects":7014,"num_object_clones":0,"num_object_copies":21042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7014,"num_write_kb":448896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658070+0000","last_change":"2024-03-04T16:11:59.935508+0000","last_active":"2024-03-04T16:14:21.658070+0000","last_peered":"2024-03-04T16:14:21.658070+0000","last_clean":"2024-03-04T16:14:21.658070+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:14:21.658070+0000","last_undegraded":"2024-03-04T16:14:21.658070+0000","last_fullsized":"2024-03-04T16:14:21.658070+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.935397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.935397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924100+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":179,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659019+0000","last_change":"2024-03-04T16:13:19.665876+0000","last_active":"2024-03-04T16:14:21.659019+0000","last_peered":"2024-03-04T16:14:21.659019+0000","last_clean":"2024-03-04T16:14:21.659019+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:14:21.659019+0000","last_undegraded":"2024-03-04T16:14:21.659019+0000","last_fullsized":"2024-03-04T16:14:21.659019+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.665604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.665604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:32.546211+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":466,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661000+0000","last_change":"2024-03-04T16:12:10.817851+0000","last_active":"2024-03-04T16:14:21.661000+0000","last_peered":"2024-03-04T16:14:21.661000+0000","last_clean":"2024-03-04T16:14:21.661000+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:14:21.661000+0000","last_undegraded":"2024-03-04T16:14:21.661000+0000","last_fullsized":"2024-03-04T16:14:21.661000+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:10.817671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:12:10.817671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":266,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660906+0000","last_change":"2024-03-04T16:13:13.647026+0000","last_active":"2024-03-04T16:14:21.660906+0000","last_peered":"2024-03-04T16:14:21.660906+0000","last_clean":"2024-03-04T16:14:21.660906+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:14:21.660906+0000","last_undegraded":"2024-03-04T16:14:21.660906+0000","last_fullsized":"2024-03-04T16:14:21.660906+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.646813+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:26.540709+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"270'7178","reported_seq":7244,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.665534+0000","last_change":"2024-03-04T16:12:58.259225+0000","last_active":"2024-03-04T16:14:22.665534+0000","last_peered":"2024-03-04T16:14:22.665534+0000","last_clean":"2024-03-04T16:14:22.665534+0000","last_became_active":"2024-03-04T16:12:58.258787+0000","last_became_peered":"2024-03-04T16:12:58.258787+0000","last_unstale":"2024-03-04T16:14:22.665534+0000","last_undegraded":"2024-03-04T16:14:22.665534+0000","last_fullsized":"2024-03-04T16:14:22.665534+0000","mapping_epoch":242,"log_start":"255'2800","ondisk_log_start":"255'2800","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:22.881529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470417408,"num_objects":7178,"num_object_clones":0,"num_object_copies":21534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7178,"num_write_kb":459392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":335,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659457+0000","last_change":"2024-03-04T16:13:53.931017+0000","last_active":"2024-03-04T16:14:21.659457+0000","last_peered":"2024-03-04T16:14:21.659457+0000","last_clean":"2024-03-04T16:14:21.659457+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:14:21.659457+0000","last_undegraded":"2024-03-04T16:14:21.659457+0000","last_fullsized":"2024-03-04T16:14:21.659457+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.930927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.930927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.204703+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":171,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652728+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:14:21.652728+0000","last_peered":"2024-03-04T16:14:21.652728+0000","last_clean":"2024-03-04T16:14:21.652728+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:14:21.652728+0000","last_undegraded":"2024-03-04T16:14:21.652728+0000","last_fullsized":"2024-03-04T16:14:21.652728+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":465,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662541+0000","last_change":"2024-03-04T16:13:07.694160+0000","last_active":"2024-03-04T16:14:21.662541+0000","last_peered":"2024-03-04T16:14:21.662541+0000","last_clean":"2024-03-04T16:14:21.662541+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:14:21.662541+0000","last_undegraded":"2024-03-04T16:14:21.662541+0000","last_fullsized":"2024-03-04T16:14:21.662541+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:07.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:13:07.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:29.644387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":315,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658902+0000","last_change":"2024-03-04T16:13:16.698881+0000","last_active":"2024-03-04T16:14:21.658902+0000","last_peered":"2024-03-04T16:14:21.658902+0000","last_clean":"2024-03-04T16:14:21.658902+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:14:21.658902+0000","last_undegraded":"2024-03-04T16:14:21.658902+0000","last_fullsized":"2024-03-04T16:14:21.658902+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.698648+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:40.220710+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"270'7176","reported_seq":7242,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.255326+0000","last_change":"2024-03-04T16:12:58.259820+0000","last_active":"2024-03-04T16:14:23.255326+0000","last_peered":"2024-03-04T16:14:23.255326+0000","last_clean":"2024-03-04T16:14:23.255326+0000","last_became_active":"2024-03-04T16:12:58.259075+0000","last_became_peered":"2024-03-04T16:12:58.259075+0000","last_unstale":"2024-03-04T16:14:23.255326+0000","last_undegraded":"2024-03-04T16:14:23.255326+0000","last_fullsized":"2024-03-04T16:14:23.255326+0000","mapping_epoch":242,"log_start":"255'3000","ondisk_log_start":"255'3000","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.412134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470286336,"num_objects":7176,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":459264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":432,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663530+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:14:21.663530+0000","last_peered":"2024-03-04T16:14:21.663530+0000","last_clean":"2024-03-04T16:14:21.663530+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:14:21.663530+0000","last_undegraded":"2024-03-04T16:14:21.663530+0000","last_fullsized":"2024-03-04T16:14:21.663530+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747461+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":480,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663772+0000","last_change":"2024-03-04T16:12:58.666189+0000","last_active":"2024-03-04T16:14:21.663772+0000","last_peered":"2024-03-04T16:14:21.663772+0000","last_clean":"2024-03-04T16:14:21.663772+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:14:21.663772+0000","last_undegraded":"2024-03-04T16:14:21.663772+0000","last_fullsized":"2024-03-04T16:14:21.663772+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:12:58.666070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:12:58.666070+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:28.210264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":478,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.653147+0000","last_change":"2024-03-04T16:12:25.567411+0000","last_active":"2024-03-04T16:14:21.653147+0000","last_peered":"2024-03-04T16:14:21.653147+0000","last_clean":"2024-03-04T16:14:21.653147+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:14:21.653147+0000","last_undegraded":"2024-03-04T16:14:21.653147+0000","last_fullsized":"2024-03-04T16:14:21.653147+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:25.567295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.567295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414909+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":710,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658911+0000","last_change":"2024-03-04T16:12:18.618110+0000","last_active":"2024-03-04T16:14:21.658911+0000","last_peered":"2024-03-04T16:14:21.658911+0000","last_clean":"2024-03-04T16:14:21.658911+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:14:21.658911+0000","last_undegraded":"2024-03-04T16:14:21.658911+0000","last_fullsized":"2024-03-04T16:14:21.658911+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'30","last_scrub_stamp":"2024-03-04T16:12:18.617922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.617922+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":296,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.665077+0000","last_change":"2024-03-04T16:13:52.478892+0000","last_active":"2024-03-04T16:14:21.665077+0000","last_peered":"2024-03-04T16:14:21.665077+0000","last_clean":"2024-03-04T16:14:21.665077+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:14:21.665077+0000","last_undegraded":"2024-03-04T16:14:21.665077+0000","last_fullsized":"2024-03-04T16:14:21.665077+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.478776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.478776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.051799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"270'7135","reported_seq":7197,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.692802+0000","last_change":"2024-03-04T16:12:58.263282+0000","last_active":"2024-03-04T16:14:22.692802+0000","last_peered":"2024-03-04T16:14:22.692802+0000","last_clean":"2024-03-04T16:14:22.692802+0000","last_became_active":"2024-03-04T16:12:58.262950+0000","last_became_peered":"2024-03-04T16:12:58.262950+0000","last_unstale":"2024-03-04T16:14:22.692802+0000","last_undegraded":"2024-03-04T16:14:22.692802+0000","last_fullsized":"2024-03-04T16:14:22.692802+0000","mapping_epoch":242,"log_start":"255'2700","ondisk_log_start":"255'2700","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467599360,"num_objects":7135,"num_object_clones":0,"num_object_copies":21405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7135,"num_write_kb":456640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":454,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661146+0000","last_change":"2024-03-04T16:13:20.064259+0000","last_active":"2024-03-04T16:14:21.661146+0000","last_peered":"2024-03-04T16:14:21.661146+0000","last_clean":"2024-03-04T16:14:21.661146+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:14:21.661146+0000","last_undegraded":"2024-03-04T16:14:21.661146+0000","last_fullsized":"2024-03-04T16:14:21.661146+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:20.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:20.064148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.461357+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":464,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.655243+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:14:21.655243+0000","last_peered":"2024-03-04T16:14:21.655243+0000","last_clean":"2024-03-04T16:14:21.655243+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:14:21.655243+0000","last_undegraded":"2024-03-04T16:14:21.655243+0000","last_fullsized":"2024-03-04T16:14:21.655243+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716262+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":180,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664406+0000","last_change":"2024-03-04T16:14:01.534096+0000","last_active":"2024-03-04T16:14:21.664406+0000","last_peered":"2024-03-04T16:14:21.664406+0000","last_clean":"2024-03-04T16:14:21.664406+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:14:21.664406+0000","last_undegraded":"2024-03-04T16:14:21.664406+0000","last_fullsized":"2024-03-04T16:14:21.664406+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.533961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.533961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:08.148103+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":95,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659237+0000","last_change":"2024-03-04T16:12:35.728287+0000","last_active":"2024-03-04T16:14:21.659237+0000","last_peered":"2024-03-04T16:14:21.659237+0000","last_clean":"2024-03-04T16:14:21.659237+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:14:21.659237+0000","last_undegraded":"2024-03-04T16:14:21.659237+0000","last_fullsized":"2024-03-04T16:14:21.659237+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:11:57.950523+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:11:57.950523+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716296+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":171,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663197+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:14:21.663197+0000","last_peered":"2024-03-04T16:14:21.663197+0000","last_clean":"2024-03-04T16:14:21.663197+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:14:21.663197+0000","last_undegraded":"2024-03-04T16:14:21.663197+0000","last_fullsized":"2024-03-04T16:14:21.663197+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"270'7185","reported_seq":7251,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.551127+0000","last_change":"2024-03-04T16:12:58.256227+0000","last_active":"2024-03-04T16:14:23.551127+0000","last_peered":"2024-03-04T16:14:23.551127+0000","last_clean":"2024-03-04T16:14:23.551127+0000","last_became_active":"2024-03-04T16:12:58.255897+0000","last_became_peered":"2024-03-04T16:12:58.255897+0000","last_unstale":"2024-03-04T16:14:23.551127+0000","last_undegraded":"2024-03-04T16:14:23.551127+0000","last_fullsized":"2024-03-04T16:14:23.551127+0000","mapping_epoch":242,"log_start":"256'3100","ondisk_log_start":"256'3100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:27.831985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470876160,"num_objects":7185,"num_object_clones":0,"num_object_copies":21555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7185,"num_write_kb":459840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":447,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664305+0000","last_change":"2024-03-04T16:12:23.816872+0000","last_active":"2024-03-04T16:14:21.664305+0000","last_peered":"2024-03-04T16:14:21.664305+0000","last_clean":"2024-03-04T16:14:21.664305+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:14:21.664305+0000","last_undegraded":"2024-03-04T16:14:21.664305+0000","last_fullsized":"2024-03-04T16:14:21.664305+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.816724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.816724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":766,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658205+0000","last_change":"2024-03-04T16:12:59.102019+0000","last_active":"2024-03-04T16:14:21.658205+0000","last_peered":"2024-03-04T16:14:21.658205+0000","last_clean":"2024-03-04T16:14:21.658205+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:14:21.658205+0000","last_undegraded":"2024-03-04T16:14:21.658205+0000","last_fullsized":"2024-03-04T16:14:21.658205+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:12:59.101913+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:12:59.101913+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663373+0000","last_change":"2024-03-04T16:13:56.375663+0000","last_active":"2024-03-04T16:14:21.663373+0000","last_peered":"2024-03-04T16:14:21.663373+0000","last_clean":"2024-03-04T16:14:21.663373+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:14:21.663373+0000","last_undegraded":"2024-03-04T16:14:21.663373+0000","last_fullsized":"2024-03-04T16:14:21.663373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.375405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.375405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.685421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":290,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652875+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:14:21.652875+0000","last_peered":"2024-03-04T16:14:21.652875+0000","last_clean":"2024-03-04T16:14:21.652875+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:14:21.652875+0000","last_undegraded":"2024-03-04T16:14:21.652875+0000","last_fullsized":"2024-03-04T16:14:21.652875+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659086+0000","last_change":"2024-03-04T16:12:09.805823+0000","last_active":"2024-03-04T16:14:21.659086+0000","last_peered":"2024-03-04T16:14:21.659086+0000","last_clean":"2024-03-04T16:14:21.659086+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:14:21.659086+0000","last_undegraded":"2024-03-04T16:14:21.659086+0000","last_fullsized":"2024-03-04T16:14:21.659086+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:09.805715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:09.805715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924209+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"270'7063","reported_seq":7129,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.262221+0000","last_change":"2024-03-04T16:12:58.258704+0000","last_active":"2024-03-04T16:14:23.262221+0000","last_peered":"2024-03-04T16:14:23.262221+0000","last_clean":"2024-03-04T16:14:23.262221+0000","last_became_active":"2024-03-04T16:12:58.258386+0000","last_became_peered":"2024-03-04T16:12:58.258386+0000","last_unstale":"2024-03-04T16:14:23.262221+0000","last_undegraded":"2024-03-04T16:14:23.262221+0000","last_fullsized":"2024-03-04T16:14:23.262221+0000","mapping_epoch":242,"log_start":"255'2900","ondisk_log_start":"255'2900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4163,"log_dups_size":0,"ondisk_log_size":4163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.412100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462880768,"num_objects":7063,"num_object_clones":0,"num_object_copies":21189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7063,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7063,"num_write_kb":452032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":448,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658325+0000","last_change":"2024-03-04T16:12:26.602465+0000","last_active":"2024-03-04T16:14:21.658325+0000","last_peered":"2024-03-04T16:14:21.658325+0000","last_clean":"2024-03-04T16:14:21.658325+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:14:21.658325+0000","last_undegraded":"2024-03-04T16:14:21.658325+0000","last_fullsized":"2024-03-04T16:14:21.658325+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:26.602350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.602350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716449+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":304,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658300+0000","last_change":"2024-03-04T16:13:50.496078+0000","last_active":"2024-03-04T16:14:21.658300+0000","last_peered":"2024-03-04T16:14:21.658300+0000","last_clean":"2024-03-04T16:14:21.658300+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:14:21.658300+0000","last_undegraded":"2024-03-04T16:14:21.658300+0000","last_fullsized":"2024-03-04T16:14:21.658300+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:50.495814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:50.495814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.556381+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1173,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658262+0000","last_change":"2024-03-04T16:13:57.825817+0000","last_active":"2024-03-04T16:14:21.658262+0000","last_peered":"2024-03-04T16:14:21.658262+0000","last_clean":"2024-03-04T16:14:21.658262+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:14:21.658262+0000","last_undegraded":"2024-03-04T16:14:21.658262+0000","last_fullsized":"2024-03-04T16:14:21.658262+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:13:57.825680+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:13:57.825680+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.150143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":692,"num_read_kb":687,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":348,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659750+0000","last_change":"2024-03-04T16:13:15.734217+0000","last_active":"2024-03-04T16:14:21.659750+0000","last_peered":"2024-03-04T16:14:21.659750+0000","last_clean":"2024-03-04T16:14:21.659750+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:14:21.659750+0000","last_undegraded":"2024-03-04T16:14:21.659750+0000","last_fullsized":"2024-03-04T16:14:21.659750+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:13:15.734074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.734074+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:45.507334+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":459,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664564+0000","last_change":"2024-03-04T16:14:06.002348+0000","last_active":"2024-03-04T16:14:21.664564+0000","last_peered":"2024-03-04T16:14:21.664564+0000","last_clean":"2024-03-04T16:14:21.664564+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:14:21.664564+0000","last_undegraded":"2024-03-04T16:14:21.664564+0000","last_fullsized":"2024-03-04T16:14:21.664564+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:06.002229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.002229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:30.238482+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"270'7040","reported_seq":7106,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.530912+0000","last_change":"2024-03-04T16:12:58.261720+0000","last_active":"2024-03-04T16:14:22.530912+0000","last_peered":"2024-03-04T16:14:22.530912+0000","last_clean":"2024-03-04T16:14:22.530912+0000","last_became_active":"2024-03-04T16:12:58.261282+0000","last_became_peered":"2024-03-04T16:12:58.261282+0000","last_unstale":"2024-03-04T16:14:22.530912+0000","last_undegraded":"2024-03-04T16:14:22.530912+0000","last_fullsized":"2024-03-04T16:14:22.530912+0000","mapping_epoch":242,"log_start":"255'2900","ondisk_log_start":"255'2900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4140,"log_dups_size":0,"ondisk_log_size":4140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.957277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461373440,"num_objects":7040,"num_object_clones":0,"num_object_copies":21120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7040,"num_write_kb":450560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.655226+0000","last_change":"2024-03-04T16:13:59.616838+0000","last_active":"2024-03-04T16:14:21.655226+0000","last_peered":"2024-03-04T16:14:21.655226+0000","last_clean":"2024-03-04T16:14:21.655226+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:14:21.655226+0000","last_undegraded":"2024-03-04T16:14:21.655226+0000","last_fullsized":"2024-03-04T16:14:21.655226+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.616709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.616709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.656637+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658744+0000","last_change":"2024-03-04T16:13:05.550936+0000","last_active":"2024-03-04T16:14:21.658744+0000","last_peered":"2024-03-04T16:14:21.658744+0000","last_clean":"2024-03-04T16:14:21.658744+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:14:21.658744+0000","last_undegraded":"2024-03-04T16:14:21.658744+0000","last_fullsized":"2024-03-04T16:14:21.658744+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.550829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.550829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:33.010264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":434,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663345+0000","last_change":"2024-03-04T16:12:17.635817+0000","last_active":"2024-03-04T16:14:21.663345+0000","last_peered":"2024-03-04T16:14:21.663345+0000","last_clean":"2024-03-04T16:14:21.663345+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:14:21.663345+0000","last_undegraded":"2024-03-04T16:14:21.663345+0000","last_fullsized":"2024-03-04T16:14:21.663345+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:12:17.635712+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:12:17.635712+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":163,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663999+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:14:21.663999+0000","last_peered":"2024-03-04T16:14:21.663999+0000","last_clean":"2024-03-04T16:14:21.663999+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:14:21.663999+0000","last_undegraded":"2024-03-04T16:14:21.663999+0000","last_fullsized":"2024-03-04T16:14:21.663999+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"270'7169","reported_seq":7235,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.670522+0000","last_change":"2024-03-04T16:12:58.262941+0000","last_active":"2024-03-04T16:14:22.670522+0000","last_peered":"2024-03-04T16:14:22.670522+0000","last_clean":"2024-03-04T16:14:22.670522+0000","last_became_active":"2024-03-04T16:12:58.262611+0000","last_became_peered":"2024-03-04T16:12:58.262611+0000","last_unstale":"2024-03-04T16:14:22.670522+0000","last_undegraded":"2024-03-04T16:14:22.670522+0000","last_fullsized":"2024-03-04T16:14:22.670522+0000","mapping_epoch":242,"log_start":"255'2800","ondisk_log_start":"255'2800","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469827584,"num_objects":7169,"num_object_clones":0,"num_object_copies":21507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7169,"num_write_kb":458816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":462,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661305+0000","last_change":"2024-03-04T16:14:01.478625+0000","last_active":"2024-03-04T16:14:21.661305+0000","last_peered":"2024-03-04T16:14:21.661305+0000","last_clean":"2024-03-04T16:14:21.661305+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:14:21.661305+0000","last_undegraded":"2024-03-04T16:14:21.661305+0000","last_fullsized":"2024-03-04T16:14:21.661305+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.477978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.477978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:23.643549+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":471,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664519+0000","last_change":"2024-03-04T16:13:16.814942+0000","last_active":"2024-03-04T16:14:21.664519+0000","last_peered":"2024-03-04T16:14:21.664519+0000","last_clean":"2024-03-04T16:14:21.664519+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:14:21.664519+0000","last_undegraded":"2024-03-04T16:14:21.664519+0000","last_fullsized":"2024-03-04T16:14:21.664519+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.814825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.814825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:46.355001+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":491,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661433+0000","last_change":"2024-03-04T16:13:16.606983+0000","last_active":"2024-03-04T16:14:21.661433+0000","last_peered":"2024-03-04T16:14:21.661433+0000","last_clean":"2024-03-04T16:14:21.661433+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:14:21.661433+0000","last_undegraded":"2024-03-04T16:14:21.661433+0000","last_fullsized":"2024-03-04T16:14:21.661433+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.605307+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.677321+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":247,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.663902+0000","last_change":"2024-03-04T16:13:58.080312+0000","last_active":"2024-03-04T16:14:21.663902+0000","last_peered":"2024-03-04T16:14:21.663902+0000","last_clean":"2024-03-04T16:14:21.663902+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:14:21.663902+0000","last_undegraded":"2024-03-04T16:14:21.663902+0000","last_fullsized":"2024-03-04T16:14:21.663902+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:13:58.080163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.080163+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:19.165616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"270'7342","reported_seq":7408,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.555093+0000","last_change":"2024-03-04T16:12:58.259682+0000","last_active":"2024-03-04T16:14:23.555093+0000","last_peered":"2024-03-04T16:14:23.555093+0000","last_clean":"2024-03-04T16:14:23.555093+0000","last_became_active":"2024-03-04T16:12:58.259080+0000","last_became_peered":"2024-03-04T16:12:58.259080+0000","last_unstale":"2024-03-04T16:14:23.555093+0000","last_undegraded":"2024-03-04T16:14:23.555093+0000","last_fullsized":"2024-03-04T16:14:23.555093+0000","mapping_epoch":242,"log_start":"257'3300","ondisk_log_start":"257'3300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4042,"log_dups_size":0,"ondisk_log_size":4042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.749709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481165312,"num_objects":7342,"num_object_clones":0,"num_object_copies":22026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7342,"num_write_kb":469888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":460,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.658954+0000","last_change":"2024-03-04T16:11:59.019162+0000","last_active":"2024-03-04T16:14:21.658954+0000","last_peered":"2024-03-04T16:14:21.658954+0000","last_clean":"2024-03-04T16:14:21.658954+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:14:21.658954+0000","last_undegraded":"2024-03-04T16:14:21.658954+0000","last_fullsized":"2024-03-04T16:14:21.658954+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.019082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.019082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.924238+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":300,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659853+0000","last_change":"2024-03-04T16:12:25.645249+0000","last_active":"2024-03-04T16:14:21.659853+0000","last_peered":"2024-03-04T16:14:21.659853+0000","last_clean":"2024-03-04T16:14:21.659853+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:14:21.659853+0000","last_undegraded":"2024-03-04T16:14:21.659853+0000","last_fullsized":"2024-03-04T16:14:21.659853+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:12:25.645114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.645114+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:24.411823+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":519,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652315+0000","last_change":"2024-03-04T16:12:24.566421+0000","last_active":"2024-03-04T16:14:21.652315+0000","last_peered":"2024-03-04T16:14:21.652315+0000","last_clean":"2024-03-04T16:14:21.652315+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:14:21.652315+0000","last_undegraded":"2024-03-04T16:14:21.652315+0000","last_fullsized":"2024-03-04T16:14:21.652315+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:12:24.566235+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.566235+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414971+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":584,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.662061+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:14:21.662061+0000","last_peered":"2024-03-04T16:14:21.662061+0000","last_clean":"2024-03-04T16:14:21.662061+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:14:21.662061+0000","last_undegraded":"2024-03-04T16:14:21.662061+0000","last_fullsized":"2024-03-04T16:14:21.662061+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"270'6969","reported_seq":7035,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.555637+0000","last_change":"2024-03-04T16:12:58.261655+0000","last_active":"2024-03-04T16:14:22.555637+0000","last_peered":"2024-03-04T16:14:22.555637+0000","last_clean":"2024-03-04T16:14:22.555637+0000","last_became_active":"2024-03-04T16:12:58.261321+0000","last_became_peered":"2024-03-04T16:12:58.261321+0000","last_unstale":"2024-03-04T16:14:22.555637+0000","last_undegraded":"2024-03-04T16:14:22.555637+0000","last_fullsized":"2024-03-04T16:14:22.555637+0000","mapping_epoch":242,"log_start":"255'2900","ondisk_log_start":"255'2900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4069,"log_dups_size":0,"ondisk_log_size":4069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456720384,"num_objects":6969,"num_object_clones":0,"num_object_copies":20907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6969,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6969,"num_write_kb":446016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652168+0000","last_change":"2024-03-04T16:13:15.438272+0000","last_active":"2024-03-04T16:14:21.652168+0000","last_peered":"2024-03-04T16:14:21.652168+0000","last_clean":"2024-03-04T16:14:21.652168+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:14:21.652168+0000","last_undegraded":"2024-03-04T16:14:21.652168+0000","last_fullsized":"2024-03-04T16:14:21.652168+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:15.438148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.438148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:30.749451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":456,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664242+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:14:21.664242+0000","last_peered":"2024-03-04T16:14:21.664242+0000","last_clean":"2024-03-04T16:14:21.664242+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:14:21.664242+0000","last_undegraded":"2024-03-04T16:14:21.664242+0000","last_fullsized":"2024-03-04T16:14:21.664242+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747489+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":546,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661958+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:14:21.661958+0000","last_peered":"2024-03-04T16:14:21.661958+0000","last_clean":"2024-03-04T16:14:21.661958+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:14:21.661958+0000","last_undegraded":"2024-03-04T16:14:21.661958+0000","last_fullsized":"2024-03-04T16:14:21.661958+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747633+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":444,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652029+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:14:21.652029+0000","last_peered":"2024-03-04T16:14:21.652029+0000","last_clean":"2024-03-04T16:14:21.652029+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:14:21.652029+0000","last_undegraded":"2024-03-04T16:14:21.652029+0000","last_fullsized":"2024-03-04T16:14:21.652029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414514+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"270'7084","reported_seq":7150,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.566820+0000","last_change":"2024-03-04T16:12:58.259473+0000","last_active":"2024-03-04T16:14:23.566820+0000","last_peered":"2024-03-04T16:14:23.566820+0000","last_clean":"2024-03-04T16:14:23.566820+0000","last_became_active":"2024-03-04T16:12:58.259164+0000","last_became_peered":"2024-03-04T16:12:58.259164+0000","last_unstale":"2024-03-04T16:14:23.566820+0000","last_undegraded":"2024-03-04T16:14:23.566820+0000","last_fullsized":"2024-03-04T16:14:23.566820+0000","mapping_epoch":242,"log_start":"255'3000","ondisk_log_start":"255'3000","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4084,"log_dups_size":0,"ondisk_log_size":4084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.749743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464257024,"num_objects":7084,"num_object_clones":0,"num_object_copies":21252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7084,"num_write_kb":453376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":451,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664305+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:14:21.664305+0000","last_peered":"2024-03-04T16:14:21.664305+0000","last_clean":"2024-03-04T16:14:21.664305+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:14:21.664305+0000","last_undegraded":"2024-03-04T16:14:21.664305+0000","last_fullsized":"2024-03-04T16:14:21.664305+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":593,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.653119+0000","last_change":"2024-03-04T16:11:59.819002+0000","last_active":"2024-03-04T16:14:21.653119+0000","last_peered":"2024-03-04T16:14:21.653119+0000","last_clean":"2024-03-04T16:14:21.653119+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:14:21.653119+0000","last_undegraded":"2024-03-04T16:14:21.653119+0000","last_fullsized":"2024-03-04T16:14:21.653119+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:11:59.818881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.818881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.661858+0000","last_change":"2024-03-04T16:13:58.414151+0000","last_active":"2024-03-04T16:14:21.661858+0000","last_peered":"2024-03-04T16:14:21.661858+0000","last_clean":"2024-03-04T16:14:21.661858+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:14:21.661858+0000","last_undegraded":"2024-03-04T16:14:21.661858+0000","last_fullsized":"2024-03-04T16:14:21.661858+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.413756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.413756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:09.771485+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"240'27","reported_seq":491,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660230+0000","last_change":"2024-03-04T16:12:24.577754+0000","last_active":"2024-03-04T16:14:21.660230+0000","last_peered":"2024-03-04T16:14:21.660230+0000","last_clean":"2024-03-04T16:14:21.660230+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:14:21.660230+0000","last_undegraded":"2024-03-04T16:14:21.660230+0000","last_fullsized":"2024-03-04T16:14:21.660230+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:24.577573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.577573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.716422+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"270'6928","reported_seq":6994,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:22.557154+0000","last_change":"2024-03-04T16:12:58.262002+0000","last_active":"2024-03-04T16:14:22.557154+0000","last_peered":"2024-03-04T16:14:22.557154+0000","last_clean":"2024-03-04T16:14:22.557154+0000","last_became_active":"2024-03-04T16:12:58.261597+0000","last_became_peered":"2024-03-04T16:12:58.261597+0000","last_unstale":"2024-03-04T16:14:22.557154+0000","last_undegraded":"2024-03-04T16:14:22.557154+0000","last_fullsized":"2024-03-04T16:14:22.557154+0000","mapping_epoch":242,"log_start":"255'2800","ondisk_log_start":"255'2800","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4128,"log_dups_size":0,"ondisk_log_size":4128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:27.001362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454033408,"num_objects":6928,"num_object_clones":0,"num_object_copies":20784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6928,"num_write_kb":443392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":443,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.667727+0000","last_change":"2024-03-04T16:12:26.381739+0000","last_active":"2024-03-04T16:14:21.667727+0000","last_peered":"2024-03-04T16:14:21.667727+0000","last_clean":"2024-03-04T16:14:21.667727+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:14:21.667727+0000","last_undegraded":"2024-03-04T16:14:21.667727+0000","last_fullsized":"2024-03-04T16:14:21.667727+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:12:26.381198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.381198+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747745+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":457,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664462+0000","last_change":"2024-03-04T16:13:17.814252+0000","last_active":"2024-03-04T16:14:21.664462+0000","last_peered":"2024-03-04T16:14:21.664462+0000","last_clean":"2024-03-04T16:14:21.664462+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:14:21.664462+0000","last_undegraded":"2024-03-04T16:14:21.664462+0000","last_fullsized":"2024-03-04T16:14:21.664462+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.814150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.814150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:39.197394+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":458,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652110+0000","last_change":"2024-03-04T16:13:13.359950+0000","last_active":"2024-03-04T16:14:21.652110+0000","last_peered":"2024-03-04T16:14:21.652110+0000","last_clean":"2024-03-04T16:14:21.652110+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:14:21.652110+0000","last_undegraded":"2024-03-04T16:14:21.652110+0000","last_fullsized":"2024-03-04T16:14:21.652110+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.359809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.359809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:39.651630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":510,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652977+0000","last_change":"2024-03-04T16:14:06.377160+0000","last_active":"2024-03-04T16:14:21.652977+0000","last_peered":"2024-03-04T16:14:21.652977+0000","last_clean":"2024-03-04T16:14:21.652977+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:14:21.652977+0000","last_undegraded":"2024-03-04T16:14:21.652977+0000","last_fullsized":"2024-03-04T16:14:21.652977+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:14:06.377028+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.377028+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.277935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"270'7031","reported_seq":7097,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.190906+0000","last_change":"2024-03-04T16:12:58.261338+0000","last_active":"2024-03-04T16:14:23.190906+0000","last_peered":"2024-03-04T16:14:23.190906+0000","last_clean":"2024-03-04T16:14:23.190906+0000","last_became_active":"2024-03-04T16:12:58.260993+0000","last_became_peered":"2024-03-04T16:12:58.260993+0000","last_unstale":"2024-03-04T16:14:23.190906+0000","last_undegraded":"2024-03-04T16:14:23.190906+0000","last_fullsized":"2024-03-04T16:14:23.190906+0000","mapping_epoch":242,"log_start":"258'3500","ondisk_log_start":"258'3500","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3531,"log_dups_size":0,"ondisk_log_size":3531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460783616,"num_objects":7031,"num_object_clones":0,"num_object_copies":21093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7031,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":185,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664079+0000","last_change":"2024-03-04T16:13:55.518681+0000","last_active":"2024-03-04T16:14:21.664079+0000","last_peered":"2024-03-04T16:14:21.664079+0000","last_clean":"2024-03-04T16:14:21.664079+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:14:21.664079+0000","last_undegraded":"2024-03-04T16:14:21.664079+0000","last_fullsized":"2024-03-04T16:14:21.664079+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:14.056169+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":452,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.659016+0000","last_change":"2024-03-04T16:13:11.618623+0000","last_active":"2024-03-04T16:14:21.659016+0000","last_peered":"2024-03-04T16:14:21.659016+0000","last_clean":"2024-03-04T16:14:21.659016+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:14:21.659016+0000","last_undegraded":"2024-03-04T16:14:21.659016+0000","last_fullsized":"2024-03-04T16:14:21.659016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:11.618510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:11.618510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:37.256516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":171,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652270+0000","last_change":"2024-03-04T16:13:16.430026+0000","last_active":"2024-03-04T16:14:21.652270+0000","last_peered":"2024-03-04T16:14:21.652270+0000","last_clean":"2024-03-04T16:14:21.652270+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:14:21.652270+0000","last_undegraded":"2024-03-04T16:14:21.652270+0000","last_fullsized":"2024-03-04T16:14:21.652270+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.429889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.429889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.304143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":553,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652668+0000","last_change":"2024-03-04T16:13:54.369721+0000","last_active":"2024-03-04T16:14:21.652668+0000","last_peered":"2024-03-04T16:14:21.652668+0000","last_clean":"2024-03-04T16:14:21.652668+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:14:21.652668+0000","last_undegraded":"2024-03-04T16:14:21.652668+0000","last_fullsized":"2024-03-04T16:14:21.652668+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:13:54.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.369577+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.639737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"270'7154","reported_seq":7220,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:23.570533+0000","last_change":"2024-03-04T16:12:58.259516+0000","last_active":"2024-03-04T16:14:23.570533+0000","last_peered":"2024-03-04T16:14:23.570533+0000","last_clean":"2024-03-04T16:14:23.570533+0000","last_became_active":"2024-03-04T16:12:58.259119+0000","last_became_peered":"2024-03-04T16:12:58.259119+0000","last_unstale":"2024-03-04T16:14:23.570533+0000","last_undegraded":"2024-03-04T16:14:23.570533+0000","last_fullsized":"2024-03-04T16:14:23.570533+0000","mapping_epoch":242,"log_start":"256'3100","ondisk_log_start":"256'3100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4054,"log_dups_size":0,"ondisk_log_size":4054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.749669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468844544,"num_objects":7154,"num_object_clones":0,"num_object_copies":21462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":457856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":446,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.653183+0000","last_change":"2024-03-04T16:12:00.783064+0000","last_active":"2024-03-04T16:14:21.653183+0000","last_peered":"2024-03-04T16:14:21.653183+0000","last_clean":"2024-03-04T16:14:21.653183+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:14:21.653183+0000","last_undegraded":"2024-03-04T16:14:21.653183+0000","last_fullsized":"2024-03-04T16:14:21.653183+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.782828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.782828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.414937+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":320,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.660903+0000","last_change":"2024-03-04T16:13:18.019746+0000","last_active":"2024-03-04T16:14:21.660903+0000","last_peered":"2024-03-04T16:14:21.660903+0000","last_clean":"2024-03-04T16:14:21.660903+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:14:21.660903+0000","last_undegraded":"2024-03-04T16:14:21.660903+0000","last_fullsized":"2024-03-04T16:14:21.660903+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:13:18.019623+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.019623+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:34.436900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":456,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.664247+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:14:21.664247+0000","last_peered":"2024-03-04T16:14:21.664247+0000","last_clean":"2024-03-04T16:14:21.664247+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:14:21.664247+0000","last_undegraded":"2024-03-04T16:14:21.664247+0000","last_fullsized":"2024-03-04T16:14:21.664247+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:25.747572+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":496,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-04T16:14:21.652684+0000","last_change":"2024-03-04T16:13:05.484192+0000","last_active":"2024-03-04T16:14:21.652684+0000","last_peered":"2024-03-04T16:14:21.652684+0000","last_clean":"2024-03-04T16:14:21.652684+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:14:21.652684+0000","last_undegraded":"2024-03-04T16:14:21.652684+0000","last_fullsized":"2024-03-04T16:14:21.652684+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:13:05.484009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.484009+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:23.415001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":7457341440,"num_objects":113790,"num_object_clones":0,"num_object_copies":341370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113790,"num_write_kb":7282560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22465609728,"data_stored":22465609728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65790,"ondisk_log_size":65790,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":687,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070608,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4880868,"kb_used_data":4484388,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":88864284,"statfs":{"total":95995035648,"available":90997026816,"internally_reserved":0,"allocated":4592013312,"data_stored":4591510943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":405797322},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364949,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575022,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5058364,"kb_used_data":4517684,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":88686788,"statfs":{"total":95995035648,"available":90815270912,"internally_reserved":0,"allocated":4626108416,"data_stored":4625588752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109356,"kb_used_data":2712876,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90635796,"statfs":{"total":95995035648,"available":92811055104,"internally_reserved":0,"allocated":2777985024,"data_stored":2777474447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411496,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149859,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4282352,"kb_used_data":3633384,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":89462800,"statfs":{"total":95995035648,"available":91609907200,"internally_reserved":0,"allocated":3720585216,"data_stored":3720077880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":664533448},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953631,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2308444,"kb_used_data":1934356,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91436708,"statfs":{"total":95995035648,"available":93631188992,"internally_reserved":0,"allocated":1980780544,"data_stored":1980292432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626586,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5000076,"kb_used_data":4485580,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":88745076,"statfs":{"total":95995035648,"available":90874957824,"internally_reserved":0,"allocated":4593233920,"data_stored":4592753776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":526645702},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2805334016,"data_stored":2805334016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4655677440,"data_stored":4655677440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3757965312,"data_stored":3757965312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1982922752,"data_stored":1982922752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4639621120,"data_stored":4639621120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4624089088,"data_stored":4624089088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:14:29.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:14:29.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:29.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:29.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:29.432 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:29 smithi038 ceph-mon[38113]: pgmap v526: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:29.432 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:29 smithi038 ceph-mon[38113]: from='client.35417 v1:172.21.15.38:0/4247101686' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:29.432 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:29 smithi038 ceph-mon[29779]: pgmap v526: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:29.432 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:29 smithi038 ceph-mon[29779]: from='client.35417 v1:172.21.15.38:0/4247101686' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:29.455 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:29.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:29.563 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 78 16 122727 122711 98.2956 88.8125 0.0108329 0.0101691 2024-03-04T16:14:29.606 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:29 smithi176 ceph-mon[33619]: pgmap v526: 148 pgs: 148 active+clean; 6.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-04T16:14:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:29 smithi176 ceph-mon[33619]: from='client.35417 v1:172.21.15.38:0/4247101686' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:29.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:29.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:29.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:29.880 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:14:29.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:29.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:29.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:29.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:14:29.968 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:14:29.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:30.009 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:14:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.026 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:30.050 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:14:30.091 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:14:30.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:30.131 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:14:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:30.171 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:14:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:14:30.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:30.307 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:30.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:30.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:30.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:30.564 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 79 16 124049 124033 98.0971 82.625 0.0164903 0.0101893 2024-03-04T16:14:30.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:30.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3518086005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:30.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:30.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:30.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3518086005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:30.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3518086005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:30.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:30.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:30.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:30.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:31.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:31.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:31.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:31.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:31.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:31.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:31.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:31.446 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:31 smithi038 ceph-mon[29779]: pgmap v527: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:31 smithi038 ceph-mon[38113]: pgmap v527: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:31.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:31.564 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:14:31.565411+0000 min lat: 0.00128089 max lat: 0.173332 avg lat: 0.0102131 2024-03-04T16:14:31.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:14:31.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 80 16 125332 125316 97.872 80.1875 0.021355 0.0102131 2024-03-04T16:14:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:31.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:31 smithi176 ceph-mon[33619]: pgmap v527: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:31.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:31.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:31.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:31.867 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:31.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:31.969 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:32.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.004 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:32.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:32.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:32.146 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:32.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:32.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:32.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:32.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 81 16 126737 126721 97.7477 87.8125 0.0160052 0.0102264 2024-03-04T16:14:32.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:32 smithi176 ceph-mon[33619]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:32.663 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:32.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:32 smithi038 ceph-mon[29779]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:32.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:32 smithi038 ceph-mon[38113]: pgmap v528: 148 pgs: 148 active+clean; 7.4 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-04T16:14:32.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:32.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.833 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:32.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:32.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:32.975 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:32.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:33.077 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.112 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:33.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:33.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:33.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:33.310 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:33.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:33.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:33.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:33.532 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:33.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:33.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 82 16 128125 128109 97.6136 86.75 0.00931793 0.0102401 2024-03-04T16:14:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:33.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.678 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:33.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:33.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:33.773 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:33.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:33.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:14:33.810 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:33.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:33.946 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:33.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:34.048 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.114 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:34.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:34.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.252 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:34.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:34.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:34.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.389 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:34.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:34.472 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:34 smithi038 ceph-mon[38113]: pgmap v529: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-04T16:14:34.472 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:34 smithi038 ceph-mon[29779]: pgmap v529: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-04T16:14:34.490 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:34.534 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:34.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 83 16 129427 129411 97.4179 81.375 0.0109488 0.0102607 2024-03-04T16:14:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:34 smithi176 ceph-mon[33619]: pgmap v529: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.3 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-04T16:14:34.636 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:34.672 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:34.775 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:34.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:34.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:34.951 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:34.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:35.052 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:35.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:35.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:35.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:35.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:35.371 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:14:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:35.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:35.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1331492227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:35 smithi038 ceph-mon[29779]: from='client.25393 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1331492227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:35 smithi038 ceph-mon[38113]: from='client.25393 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.524 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:35.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 84 16 130859 130843 97.3235 89.5 0.00785819 0.0102707 2024-03-04T16:14:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1331492227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:35 smithi176 ceph-mon[33619]: from='client.25393 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:14:35.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:35.661 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:35.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:35.801 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:35.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:35.940 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:35.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:36.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:36.079 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:36.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:36.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:36.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:36.379 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:36.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: from='client.25393 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: osdmap e271: 8 total, 8 up, 6 in 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: pgmap v531: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: from='client.25393 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:14:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: osdmap e271: 8 total, 8 up, 6 in 2024-03-04T16:14:36.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: pgmap v531: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:36.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:14:36.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:36 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:14:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.521 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:36.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:36.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 85 16 132159 132143 97.1343 81.25 0.00280037 0.0102854 2024-03-04T16:14:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:14:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: from='client.25393 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:14:36.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: osdmap e271: 8 total, 8 up, 6 in 2024-03-04T16:14:36.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: pgmap v531: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1023 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:14:36.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:14:36.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:36 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:14:36.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:36.652 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:36.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:36.787 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:36.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:36.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:36.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:37.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:37.064 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:37.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:37.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:37.206 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:37.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:37.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:37.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:37.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:14:37.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:37 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:14:37.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 86 16 133297 133281 96.8319 71.125 0.00293892 0.0103128 2024-03-04T16:14:37.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:37.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:37.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:37.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:37.753 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:37.800 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:37.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:37 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:14:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:37 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:14:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:37 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:14:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:37 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:14:37.883 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:37.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:37.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:38.021 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:38.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:38.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:38.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:38.318 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:38.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:38.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:38.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: osdmap e272: 8 total, 8 up, 6 in 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: pgmap v533: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:14:38.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: osdmap e272: 8 total, 8 up, 6 in 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: pgmap v533: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:14:38.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:38.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:38.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 87 16 134519 134503 96.5968 76.375 0.0102877 0.0103479 2024-03-04T16:14:38.585 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:38.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:38.727 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:38.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: osdmap e272: 8 total, 8 up, 6 in 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: pgmap v533: 148 pgs: 148 active+clean; 7.8 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 847 op/s 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:14:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:38.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:38.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:38.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:39.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:39.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:39.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.141 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:39.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:39.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:39.278 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:39.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:39.425 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:39.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:39.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:14:39.564 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:39.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 88 16 135862 135846 96.4529 83.9375 0.0169334 0.0103633 2024-03-04T16:14:39.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:39 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:14:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:39 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:14:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:39 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:14:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:39 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:14:39.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:39.823 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:14:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:39.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:39.829 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:14:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:39 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:14:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:39 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:14:39.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:39.987 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:39.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:40.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:40.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:40.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:40.133 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:14:40.133 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:14:40.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.262 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:40.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:40.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:40.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:40.565 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 89 16 137075 137059 96.2208 75.8125 0.0050764 0.0103884 2024-03-04T16:14:40.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:40.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:40.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: pgmap v534: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: from='client.25396 v1:172.21.15.38:0/1451633273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:14:40.658 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:14:40.659 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: pgmap v534: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:14:40.659 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: from='client.25396 v1:172.21.15.38:0/1451633273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:40.659 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:14:40.659 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:40 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:14:40.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:40.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:40.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: pgmap v534: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: from='client.25396 v1:172.21.15.38:0/1451633273' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:14:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:40 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:14:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:40.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:40.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:40.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.031 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:41.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:41.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:41.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:41.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:41.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.322 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:41.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:41.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.467 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:41.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total time run: 90.0099 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total writes made: 138488 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Write size: 65536 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object size: 65536 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Bandwidth (MB/sec): 96.1616 2024-03-04T16:14:41.566 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Bandwidth: 28.8634 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max bandwidth (MB/sec): 200.125 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min bandwidth (MB/sec): 60.875 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average IOPS: 1538 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev IOPS: 461.815 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max IOPS: 3202 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min IOPS: 974 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average Latency(s): 0.0103951 2024-03-04T16:14:41.567 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Latency(s): 0.0080785 2024-03-04T16:14:41.568 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max latency(s): 0.173332 2024-03-04T16:14:41.568 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min latency(s): 0.00128089 2024-03-04T16:14:41.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:41.571 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Cleaning up (deleting benchmark objects) 2024-03-04T16:14:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:41.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:41.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:41.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:41 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:14:41.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:41 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:14:41.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:41 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:14:41.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:41 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:14:41.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:41.829 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2024-03-04T16:14:39.535076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8777177438,"num_objects":134118,"num_object_clones":0,"num_object_copies":402354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134118,"num_whiteouts":0,"num_read":2055,"num_read_kb":1901,"num_write":134821,"num_write_kb":8573341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66674,"ondisk_log_size":66674,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":28503076,"kb_used_data":25553404,"kb_used_omap":9,"kb_used_meta":2948342,"kb_avail":533967836,"statfs":{"total":575970213888,"available":546783064064,"internally_reserved":0,"allocated":26166685696,"data_stored":26164237912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":3019102887},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":869859328,"num_objects":13273,"num_object_clones":0,"num_object_copies":39819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13273,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13273,"num_write_kb":849472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001895"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":451,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397479+0000","last_change":"2024-03-04T16:13:18.470267+0000","last_active":"2024-03-04T16:14:37.397479+0000","last_peered":"2024-03-04T16:14:37.397479+0000","last_clean":"2024-03-04T16:14:37.397479+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:14:37.397479+0000","last_undegraded":"2024-03-04T16:14:37.397479+0000","last_fullsized":"2024-03-04T16:14:37.397479+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:18.470133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.470133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:48.009404+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":447,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.392671+0000","last_change":"2024-03-04T16:12:19.579690+0000","last_active":"2024-03-04T16:14:37.392671+0000","last_peered":"2024-03-04T16:14:37.392671+0000","last_clean":"2024-03-04T16:14:37.392671+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:14:37.392671+0000","last_undegraded":"2024-03-04T16:14:37.392671+0000","last_fullsized":"2024-03-04T16:14:37.392671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.579557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.579557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":312,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386561+0000","last_change":"2024-03-04T16:13:53.441131+0000","last_active":"2024-03-04T16:14:37.386561+0000","last_peered":"2024-03-04T16:14:37.386561+0000","last_clean":"2024-03-04T16:14:37.386561+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:14:37.386561+0000","last_undegraded":"2024-03-04T16:14:37.386561+0000","last_fullsized":"2024-03-04T16:14:37.386561+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.441019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.441019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.500140+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":492,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397055+0000","last_change":"2024-03-04T16:10:29.346143+0000","last_active":"2024-03-04T16:14:37.397055+0000","last_peered":"2024-03-04T16:14:37.397055+0000","last_clean":"2024-03-04T16:14:37.397055+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:14:37.397055+0000","last_undegraded":"2024-03-04T16:14:37.397055+0000","last_fullsized":"2024-03-04T16:14:37.397055+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'9","last_scrub_stamp":"2024-03-04T16:10:13.909962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:13.909962+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":441,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387084+0000","last_change":"2024-03-04T16:10:29.247209+0000","last_active":"2024-03-04T16:14:37.387084+0000","last_peered":"2024-03-04T16:14:37.387084+0000","last_clean":"2024-03-04T16:14:37.387084+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:14:37.387084+0000","last_undegraded":"2024-03-04T16:14:37.387084+0000","last_fullsized":"2024-03-04T16:14:37.387084+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:20.901980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:09:20.901980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766460+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":453,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397165+0000","last_change":"2024-03-04T16:12:14.551847+0000","last_active":"2024-03-04T16:14:37.397165+0000","last_peered":"2024-03-04T16:14:37.397165+0000","last_clean":"2024-03-04T16:14:37.397165+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:14:37.397165+0000","last_undegraded":"2024-03-04T16:14:37.397165+0000","last_fullsized":"2024-03-04T16:14:37.397165+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.551417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.551417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":460,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.382082+0000","last_change":"2024-03-04T16:13:10.472225+0000","last_active":"2024-03-04T16:14:37.382082+0000","last_peered":"2024-03-04T16:14:37.382082+0000","last_clean":"2024-03-04T16:14:37.382082+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:14:37.382082+0000","last_undegraded":"2024-03-04T16:14:37.382082+0000","last_fullsized":"2024-03-04T16:14:37.382082+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:10.471872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:13:10.471872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":560,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.384819+0000","last_change":"2024-03-04T16:12:22.299425+0000","last_active":"2024-03-04T16:14:37.384819+0000","last_peered":"2024-03-04T16:14:37.384819+0000","last_clean":"2024-03-04T16:14:37.384819+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:14:37.384819+0000","last_undegraded":"2024-03-04T16:14:37.384819+0000","last_fullsized":"2024-03-04T16:14:37.384819+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:22.299279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.299279+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":307,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388704+0000","last_change":"2024-03-04T16:14:35.605052+0000","last_active":"2024-03-04T16:14:37.388704+0000","last_peered":"2024-03-04T16:14:37.388704+0000","last_clean":"2024-03-04T16:14:37.388704+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:14:37.388704+0000","last_undegraded":"2024-03-04T16:14:37.388704+0000","last_fullsized":"2024-03-04T16:14:37.388704+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:35.604922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:14:35.604922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:35.945103+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":446,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380089+0000","last_change":"2024-03-04T16:12:13.585218+0000","last_active":"2024-03-04T16:14:37.380089+0000","last_peered":"2024-03-04T16:14:37.380089+0000","last_clean":"2024-03-04T16:14:37.380089+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:14:37.380089+0000","last_undegraded":"2024-03-04T16:14:37.380089+0000","last_fullsized":"2024-03-04T16:14:37.380089+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:13.585073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:12:13.585073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":468,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380786+0000","last_change":"2024-03-04T16:12:02.625596+0000","last_active":"2024-03-04T16:14:37.380786+0000","last_peered":"2024-03-04T16:14:37.380786+0000","last_clean":"2024-03-04T16:14:37.380786+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:14:37.380786+0000","last_undegraded":"2024-03-04T16:14:37.380786+0000","last_fullsized":"2024-03-04T16:14:37.380786+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.625486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.625486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":512,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397748+0000","last_change":"2024-03-04T16:11:58.878158+0000","last_active":"2024-03-04T16:14:37.397748+0000","last_peered":"2024-03-04T16:14:37.397748+0000","last_clean":"2024-03-04T16:14:37.397748+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:14:37.397748+0000","last_undegraded":"2024-03-04T16:14:37.397748+0000","last_fullsized":"2024-03-04T16:14:37.397748+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:11:58.878047+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:11:58.878047+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":458,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.382000+0000","last_change":"2024-03-04T16:13:59.118849+0000","last_active":"2024-03-04T16:14:37.382000+0000","last_peered":"2024-03-04T16:14:37.382000+0000","last_clean":"2024-03-04T16:14:37.382000+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:14:37.382000+0000","last_undegraded":"2024-03-04T16:14:37.382000+0000","last_fullsized":"2024-03-04T16:14:37.382000+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.118720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.495900+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":455,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386976+0000","last_change":"2024-03-04T16:12:07.816272+0000","last_active":"2024-03-04T16:14:37.386976+0000","last_peered":"2024-03-04T16:14:37.386976+0000","last_clean":"2024-03-04T16:14:37.386976+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:14:37.386976+0000","last_undegraded":"2024-03-04T16:14:37.386976+0000","last_fullsized":"2024-03-04T16:14:37.386976+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:07.816157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:07.816157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380275+0000","last_change":"2024-03-04T16:13:56.116452+0000","last_active":"2024-03-04T16:14:37.380275+0000","last_peered":"2024-03-04T16:14:37.380275+0000","last_clean":"2024-03-04T16:14:37.380275+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:14:37.380275+0000","last_undegraded":"2024-03-04T16:14:37.380275+0000","last_fullsized":"2024-03-04T16:14:37.380275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.116341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.116341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:58.120504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":490,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398556+0000","last_change":"2024-03-04T16:12:15.539669+0000","last_active":"2024-03-04T16:14:37.398556+0000","last_peered":"2024-03-04T16:14:37.398556+0000","last_clean":"2024-03-04T16:14:37.398556+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:14:37.398556+0000","last_undegraded":"2024-03-04T16:14:37.398556+0000","last_fullsized":"2024-03-04T16:14:37.398556+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.539528+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":450,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399577+0000","last_change":"2024-03-04T16:12:20.525886+0000","last_active":"2024-03-04T16:14:37.399577+0000","last_peered":"2024-03-04T16:14:37.399577+0000","last_clean":"2024-03-04T16:14:37.399577+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:14:37.399577+0000","last_undegraded":"2024-03-04T16:14:37.399577+0000","last_fullsized":"2024-03-04T16:14:37.399577+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:20.525761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.525761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":455,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.391415+0000","last_change":"2024-03-04T16:13:03.552450+0000","last_active":"2024-03-04T16:14:37.391415+0000","last_peered":"2024-03-04T16:14:37.391415+0000","last_clean":"2024-03-04T16:14:37.391415+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:14:37.391415+0000","last_undegraded":"2024-03-04T16:14:37.391415+0000","last_fullsized":"2024-03-04T16:14:37.391415+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:03.552289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:03.552289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":299,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390113+0000","last_change":"2024-03-04T16:14:36.635399+0000","last_active":"2024-03-04T16:14:37.390113+0000","last_peered":"2024-03-04T16:14:37.390113+0000","last_clean":"2024-03-04T16:14:37.390113+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:14:37.390113+0000","last_undegraded":"2024-03-04T16:14:37.390113+0000","last_fullsized":"2024-03-04T16:14:37.390113+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:36.635123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:36.635123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:37.136584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'28","reported_seq":270,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390330+0000","last_change":"2024-03-04T16:14:36.667204+0000","last_active":"2024-03-04T16:14:37.390330+0000","last_peered":"2024-03-04T16:14:37.390330+0000","last_clean":"2024-03-04T16:14:37.390330+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:14:37.390330+0000","last_undegraded":"2024-03-04T16:14:37.390330+0000","last_fullsized":"2024-03-04T16:14:37.390330+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-03-04T16:14:36.666970+0000","last_deep_scrub":"250'28","last_deep_scrub_stamp":"2024-03-04T16:13:18.621837+0000","last_clean_scrub_stamp":"2024-03-04T16:14:36.666970+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:56.196752+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":442,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388037+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:14:37.388037+0000","last_peered":"2024-03-04T16:14:37.388037+0000","last_clean":"2024-03-04T16:14:37.388037+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:14:37.388037+0000","last_undegraded":"2024-03-04T16:14:37.388037+0000","last_fullsized":"2024-03-04T16:14:37.388037+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766660+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":461,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398915+0000","last_change":"2024-03-04T16:13:58.335444+0000","last_active":"2024-03-04T16:14:37.398915+0000","last_peered":"2024-03-04T16:14:37.398915+0000","last_clean":"2024-03-04T16:14:37.398915+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:14:37.398915+0000","last_undegraded":"2024-03-04T16:14:37.398915+0000","last_fullsized":"2024-03-04T16:14:37.398915+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.334939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.334939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.215128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":469,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397786+0000","last_change":"2024-03-04T16:13:17.426445+0000","last_active":"2024-03-04T16:14:37.397786+0000","last_peered":"2024-03-04T16:14:37.397786+0000","last_clean":"2024-03-04T16:14:37.397786+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:14:37.397786+0000","last_undegraded":"2024-03-04T16:14:37.397786+0000","last_fullsized":"2024-03-04T16:14:37.397786+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.426276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.426276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:40.807189+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":483,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381496+0000","last_change":"2024-03-04T16:13:05.468168+0000","last_active":"2024-03-04T16:14:37.381496+0000","last_peered":"2024-03-04T16:14:37.381496+0000","last_clean":"2024-03-04T16:14:37.381496+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:14:37.381496+0000","last_undegraded":"2024-03-04T16:14:37.381496+0000","last_fullsized":"2024-03-04T16:14:37.381496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:13:05.468069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.468069+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":463,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386488+0000","last_change":"2024-03-04T16:13:55.908132+0000","last_active":"2024-03-04T16:14:37.386488+0000","last_peered":"2024-03-04T16:14:37.386488+0000","last_clean":"2024-03-04T16:14:37.386488+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:14:37.386488+0000","last_undegraded":"2024-03-04T16:14:37.386488+0000","last_fullsized":"2024-03-04T16:14:37.386488+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:13:55.907870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.907870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.802762+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":415,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399488+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:14:37.399488+0000","last_peered":"2024-03-04T16:14:37.399488+0000","last_clean":"2024-03-04T16:14:37.399488+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:14:37.399488+0000","last_undegraded":"2024-03-04T16:14:37.399488+0000","last_fullsized":"2024-03-04T16:14:37.399488+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.392815+0000","last_change":"2024-03-04T16:12:12.526320+0000","last_active":"2024-03-04T16:14:37.392815+0000","last_peered":"2024-03-04T16:14:37.392815+0000","last_clean":"2024-03-04T16:14:37.392815+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:14:37.392815+0000","last_undegraded":"2024-03-04T16:14:37.392815+0000","last_fullsized":"2024-03-04T16:14:37.392815+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:12:12.526200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:12.526200+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636499+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":327,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385978+0000","last_change":"2024-03-04T16:13:10.705990+0000","last_active":"2024-03-04T16:14:37.385978+0000","last_peered":"2024-03-04T16:14:37.385978+0000","last_clean":"2024-03-04T16:14:37.385978+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:14:37.385978+0000","last_undegraded":"2024-03-04T16:14:37.385978+0000","last_fullsized":"2024-03-04T16:14:37.385978+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:13:10.705885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:10.705885+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":443,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.391556+0000","last_change":"2024-03-04T16:12:23.777697+0000","last_active":"2024-03-04T16:14:37.391556+0000","last_peered":"2024-03-04T16:14:37.391556+0000","last_clean":"2024-03-04T16:14:37.391556+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:14:37.391556+0000","last_undegraded":"2024-03-04T16:14:37.391556+0000","last_fullsized":"2024-03-04T16:14:37.391556+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.777579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.777579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636466+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381864+0000","last_change":"2024-03-04T16:12:23.623101+0000","last_active":"2024-03-04T16:14:37.381864+0000","last_peered":"2024-03-04T16:14:37.381864+0000","last_clean":"2024-03-04T16:14:37.381864+0000","last_became_active":"2024-03-04T16:12:23.622602+0000","last_became_peered":"2024-03-04T16:12:23.622602+0000","last_unstale":"2024-03-04T16:14:37.381864+0000","last_undegraded":"2024-03-04T16:14:37.381864+0000","last_fullsized":"2024-03-04T16:14:37.381864+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.487604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.487604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":467,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.383421+0000","last_change":"2024-03-04T16:13:19.603451+0000","last_active":"2024-03-04T16:14:37.383421+0000","last_peered":"2024-03-04T16:14:37.383421+0000","last_clean":"2024-03-04T16:14:37.383421+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:14:37.383421+0000","last_undegraded":"2024-03-04T16:14:37.383421+0000","last_fullsized":"2024-03-04T16:14:37.383421+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.603191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.603191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":237,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381775+0000","last_change":"2024-03-04T16:12:22.748187+0000","last_active":"2024-03-04T16:14:37.381775+0000","last_peered":"2024-03-04T16:14:37.381775+0000","last_clean":"2024-03-04T16:14:37.381775+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:14:37.381775+0000","last_undegraded":"2024-03-04T16:14:37.381775+0000","last_fullsized":"2024-03-04T16:14:37.381775+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:12:22.748049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.748049+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":458,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387111+0000","last_change":"2024-03-04T16:12:01.877823+0000","last_active":"2024-03-04T16:14:37.387111+0000","last_peered":"2024-03-04T16:14:37.387111+0000","last_clean":"2024-03-04T16:14:37.387111+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:14:37.387111+0000","last_undegraded":"2024-03-04T16:14:37.387111+0000","last_fullsized":"2024-03-04T16:14:37.387111+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:01.877704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:12:01.877704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":175,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397202+0000","last_change":"2024-03-04T16:13:12.406817+0000","last_active":"2024-03-04T16:14:37.397202+0000","last_peered":"2024-03-04T16:14:37.397202+0000","last_clean":"2024-03-04T16:14:37.397202+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:14:37.397202+0000","last_undegraded":"2024-03-04T16:14:37.397202+0000","last_fullsized":"2024-03-04T16:14:37.397202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.406690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.406690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":299,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386450+0000","last_change":"2024-03-04T16:14:05.418615+0000","last_active":"2024-03-04T16:14:37.386450+0000","last_peered":"2024-03-04T16:14:37.386450+0000","last_clean":"2024-03-04T16:14:37.386450+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:14:37.386450+0000","last_undegraded":"2024-03-04T16:14:37.386450+0000","last_fullsized":"2024-03-04T16:14:37.386450+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:05.418476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:05.418476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.527281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":468,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.383506+0000","last_change":"2024-03-04T16:10:29.338500+0000","last_active":"2024-03-04T16:14:37.383506+0000","last_peered":"2024-03-04T16:14:37.383506+0000","last_clean":"2024-03-04T16:14:37.383506+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:14:37.383506+0000","last_undegraded":"2024-03-04T16:14:37.383506+0000","last_fullsized":"2024-03-04T16:14:37.383506+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'4","last_scrub_stamp":"2024-03-04T16:09:57.979645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:09:57.979645+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":454,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385367+0000","last_change":"2024-03-04T16:12:14.287844+0000","last_active":"2024-03-04T16:14:37.385367+0000","last_peered":"2024-03-04T16:14:37.385367+0000","last_clean":"2024-03-04T16:14:37.385367+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:14:37.385367+0000","last_undegraded":"2024-03-04T16:14:37.385367+0000","last_fullsized":"2024-03-04T16:14:37.385367+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.287598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766800+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386302+0000","last_change":"2024-03-04T16:13:06.136903+0000","last_active":"2024-03-04T16:14:37.386302+0000","last_peered":"2024-03-04T16:14:37.386302+0000","last_clean":"2024-03-04T16:14:37.386302+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:14:37.386302+0000","last_undegraded":"2024-03-04T16:14:37.386302+0000","last_fullsized":"2024-03-04T16:14:37.386302+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.136769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.136769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:37.985366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380873+0000","last_change":"2024-03-04T16:12:15.574564+0000","last_active":"2024-03-04T16:14:37.380873+0000","last_peered":"2024-03-04T16:14:37.380873+0000","last_clean":"2024-03-04T16:14:37.380873+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:14:37.380873+0000","last_undegraded":"2024-03-04T16:14:37.380873+0000","last_fullsized":"2024-03-04T16:14:37.380873+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:15.574491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.574491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":493,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.384716+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:14:37.384716+0000","last_peered":"2024-03-04T16:14:37.384716+0000","last_clean":"2024-03-04T16:14:37.384716+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:14:37.384716+0000","last_undegraded":"2024-03-04T16:14:37.384716+0000","last_fullsized":"2024-03-04T16:14:37.384716+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766717+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":292,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388823+0000","last_change":"2024-03-04T16:12:19.525429+0000","last_active":"2024-03-04T16:14:37.388823+0000","last_peered":"2024-03-04T16:14:37.388823+0000","last_clean":"2024-03-04T16:14:37.388823+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:14:37.388823+0000","last_undegraded":"2024-03-04T16:14:37.388823+0000","last_fullsized":"2024-03-04T16:14:37.388823+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.525317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.525317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":458,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385082+0000","last_change":"2024-03-04T16:13:04.431970+0000","last_active":"2024-03-04T16:14:37.385082+0000","last_peered":"2024-03-04T16:14:37.385082+0000","last_clean":"2024-03-04T16:14:37.385082+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:14:37.385082+0000","last_undegraded":"2024-03-04T16:14:37.385082+0000","last_fullsized":"2024-03-04T16:14:37.385082+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.431851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.431851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.775474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":463,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387616+0000","last_change":"2024-03-04T16:13:05.128608+0000","last_active":"2024-03-04T16:14:37.387616+0000","last_peered":"2024-03-04T16:14:37.387616+0000","last_clean":"2024-03-04T16:14:37.387616+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:14:37.387616+0000","last_undegraded":"2024-03-04T16:14:37.387616+0000","last_fullsized":"2024-03-04T16:14:37.387616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.128508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.128508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":521,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.391338+0000","last_change":"2024-03-04T16:13:14.622077+0000","last_active":"2024-03-04T16:14:37.391338+0000","last_peered":"2024-03-04T16:14:37.391338+0000","last_clean":"2024-03-04T16:14:37.391338+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:14:37.391338+0000","last_undegraded":"2024-03-04T16:14:37.391338+0000","last_fullsized":"2024-03-04T16:14:37.391338+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:14.621919+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.621919+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636769+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":448,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.383523+0000","last_change":"2024-03-04T16:13:13.530157+0000","last_active":"2024-03-04T16:14:37.383523+0000","last_peered":"2024-03-04T16:14:37.383523+0000","last_clean":"2024-03-04T16:14:37.383523+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:14:37.383523+0000","last_undegraded":"2024-03-04T16:14:37.383523+0000","last_fullsized":"2024-03-04T16:14:37.383523+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.530037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.394558+0000","last_change":"2024-03-04T16:13:12.612940+0000","last_active":"2024-03-04T16:14:37.394558+0000","last_peered":"2024-03-04T16:14:37.394558+0000","last_clean":"2024-03-04T16:14:37.394558+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:14:37.394558+0000","last_undegraded":"2024-03-04T16:14:37.394558+0000","last_fullsized":"2024-03-04T16:14:37.394558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.612806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.612806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:37.607460+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":470,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398820+0000","last_change":"2024-03-04T16:13:51.392879+0000","last_active":"2024-03-04T16:14:37.398820+0000","last_peered":"2024-03-04T16:14:37.398820+0000","last_clean":"2024-03-04T16:14:37.398820+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:14:37.398820+0000","last_undegraded":"2024-03-04T16:14:37.398820+0000","last_fullsized":"2024-03-04T16:14:37.398820+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:51.392701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:51.392701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.094055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":509,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398408+0000","last_change":"2024-03-04T16:13:01.422169+0000","last_active":"2024-03-04T16:14:37.398408+0000","last_peered":"2024-03-04T16:14:37.398408+0000","last_clean":"2024-03-04T16:14:37.398408+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:14:37.398408+0000","last_undegraded":"2024-03-04T16:14:37.398408+0000","last_fullsized":"2024-03-04T16:14:37.398408+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:01.422054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:01.422054+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":325,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386481+0000","last_change":"2024-03-04T16:12:03.904682+0000","last_active":"2024-03-04T16:14:37.386481+0000","last_peered":"2024-03-04T16:14:37.386481+0000","last_clean":"2024-03-04T16:14:37.386481+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:14:37.386481+0000","last_undegraded":"2024-03-04T16:14:37.386481+0000","last_fullsized":"2024-03-04T16:14:37.386481+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:03.903765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.903765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":619,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380871+0000","last_change":"2024-03-04T16:12:06.530577+0000","last_active":"2024-03-04T16:14:37.380871+0000","last_peered":"2024-03-04T16:14:37.380871+0000","last_clean":"2024-03-04T16:14:37.380871+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:14:37.380871+0000","last_undegraded":"2024-03-04T16:14:37.380871+0000","last_fullsized":"2024-03-04T16:14:37.380871+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:12:06.530482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:06.530482+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":498,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386233+0000","last_change":"2024-03-04T16:13:54.812148+0000","last_active":"2024-03-04T16:14:37.386233+0000","last_peered":"2024-03-04T16:14:37.386233+0000","last_clean":"2024-03-04T16:14:37.386233+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:14:37.386233+0000","last_undegraded":"2024-03-04T16:14:37.386233+0000","last_fullsized":"2024-03-04T16:14:37.386233+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:54.812003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.812003+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:21.284093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":510,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388396+0000","last_change":"2024-03-04T16:14:03.533418+0000","last_active":"2024-03-04T16:14:37.388396+0000","last_peered":"2024-03-04T16:14:37.388396+0000","last_clean":"2024-03-04T16:14:37.388396+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:14:37.388396+0000","last_undegraded":"2024-03-04T16:14:37.388396+0000","last_fullsized":"2024-03-04T16:14:37.388396+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:14:03.533128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:03.533128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:27.853629+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":449,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399646+0000","last_change":"2024-03-04T16:12:04.823911+0000","last_active":"2024-03-04T16:14:37.399646+0000","last_peered":"2024-03-04T16:14:37.399646+0000","last_clean":"2024-03-04T16:14:37.399646+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:14:37.399646+0000","last_undegraded":"2024-03-04T16:14:37.399646+0000","last_fullsized":"2024-03-04T16:14:37.399646+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.823773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.823773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":602,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388725+0000","last_change":"2024-03-04T16:12:20.287137+0000","last_active":"2024-03-04T16:14:37.388725+0000","last_peered":"2024-03-04T16:14:37.388725+0000","last_clean":"2024-03-04T16:14:37.388725+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:14:37.388725+0000","last_undegraded":"2024-03-04T16:14:37.388725+0000","last_fullsized":"2024-03-04T16:14:37.388725+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:12:20.286793+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.286793+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766837+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":357,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386814+0000","last_change":"2024-03-04T16:13:14.023544+0000","last_active":"2024-03-04T16:14:37.386814+0000","last_peered":"2024-03-04T16:14:37.386814+0000","last_clean":"2024-03-04T16:14:37.386814+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:14:37.386814+0000","last_undegraded":"2024-03-04T16:14:37.386814+0000","last_fullsized":"2024-03-04T16:14:37.386814+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:14.023420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.023420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":483,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381085+0000","last_change":"2024-03-04T16:13:05.639843+0000","last_active":"2024-03-04T16:14:37.381085+0000","last_peered":"2024-03-04T16:14:37.381085+0000","last_clean":"2024-03-04T16:14:37.381085+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:14:37.381085+0000","last_undegraded":"2024-03-04T16:14:37.381085+0000","last_fullsized":"2024-03-04T16:14:37.381085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:05.639731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.639731+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":175,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398996+0000","last_change":"2024-03-04T16:10:17.462855+0000","last_active":"2024-03-04T16:14:37.398996+0000","last_peered":"2024-03-04T16:14:37.398996+0000","last_clean":"2024-03-04T16:14:37.398996+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:14:37.398996+0000","last_undegraded":"2024-03-04T16:14:37.398996+0000","last_fullsized":"2024-03-04T16:14:37.398996+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:03.969606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.969606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":439,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387921+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:14:37.387921+0000","last_peered":"2024-03-04T16:14:37.387921+0000","last_clean":"2024-03-04T16:14:37.387921+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:14:37.387921+0000","last_undegraded":"2024-03-04T16:14:37.387921+0000","last_fullsized":"2024-03-04T16:14:37.387921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766551+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389828+0000","last_change":"2024-03-04T16:12:00.921309+0000","last_active":"2024-03-04T16:14:37.389828+0000","last_peered":"2024-03-04T16:14:37.389828+0000","last_clean":"2024-03-04T16:14:37.389828+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:14:37.389828+0000","last_undegraded":"2024-03-04T16:14:37.389828+0000","last_fullsized":"2024-03-04T16:14:37.389828+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.921174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.921174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":318,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386084+0000","last_change":"2024-03-04T16:13:09.675634+0000","last_active":"2024-03-04T16:14:37.386084+0000","last_peered":"2024-03-04T16:14:37.386084+0000","last_clean":"2024-03-04T16:14:37.386084+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:14:37.386084+0000","last_undegraded":"2024-03-04T16:14:37.386084+0000","last_fullsized":"2024-03-04T16:14:37.386084+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:09.675534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:09.675534+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":457,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387166+0000","last_change":"2024-03-04T16:12:18.923785+0000","last_active":"2024-03-04T16:14:37.387166+0000","last_peered":"2024-03-04T16:14:37.387166+0000","last_clean":"2024-03-04T16:14:37.387166+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:14:37.387166+0000","last_undegraded":"2024-03-04T16:14:37.387166+0000","last_fullsized":"2024-03-04T16:14:37.387166+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:18.923671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.923671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":454,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380780+0000","last_change":"2024-03-04T16:13:06.668331+0000","last_active":"2024-03-04T16:14:37.380780+0000","last_peered":"2024-03-04T16:14:37.380780+0000","last_clean":"2024-03-04T16:14:37.380780+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:14:37.380780+0000","last_undegraded":"2024-03-04T16:14:37.380780+0000","last_fullsized":"2024-03-04T16:14:37.380780+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.668128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.668128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":361,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389049+0000","last_change":"2024-03-04T16:13:03.675180+0000","last_active":"2024-03-04T16:14:37.389049+0000","last_peered":"2024-03-04T16:14:37.389049+0000","last_clean":"2024-03-04T16:14:37.389049+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:14:37.389049+0000","last_undegraded":"2024-03-04T16:14:37.389049+0000","last_fullsized":"2024-03-04T16:14:37.389049+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:03.675052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:13:03.675052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":464,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388138+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:14:37.388138+0000","last_peered":"2024-03-04T16:14:37.388138+0000","last_clean":"2024-03-04T16:14:37.388138+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:14:37.388138+0000","last_undegraded":"2024-03-04T16:14:37.388138+0000","last_fullsized":"2024-03-04T16:14:37.388138+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"272'8486","reported_seq":8556,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.280388+0000","last_change":"2024-03-04T16:12:58.261525+0000","last_active":"2024-03-04T16:14:38.280388+0000","last_peered":"2024-03-04T16:14:38.280388+0000","last_clean":"2024-03-04T16:14:38.280388+0000","last_became_active":"2024-03-04T16:12:58.261143+0000","last_became_peered":"2024-03-04T16:12:58.261143+0000","last_unstale":"2024-03-04T16:14:38.280388+0000","last_undegraded":"2024-03-04T16:14:38.280388+0000","last_fullsized":"2024-03-04T16:14:38.280388+0000","mapping_epoch":242,"log_start":"261'4300","ondisk_log_start":"261'4300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":556138496,"num_objects":8486,"num_object_clones":0,"num_object_copies":25458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":543104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":457,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380572+0000","last_change":"2024-03-04T16:13:04.643016+0000","last_active":"2024-03-04T16:14:37.380572+0000","last_peered":"2024-03-04T16:14:37.380572+0000","last_clean":"2024-03-04T16:14:37.380572+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:14:37.380572+0000","last_undegraded":"2024-03-04T16:14:37.380572+0000","last_fullsized":"2024-03-04T16:14:37.380572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.642875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.642875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":500,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387218+0000","last_change":"2024-03-04T16:12:26.939744+0000","last_active":"2024-03-04T16:14:37.387218+0000","last_peered":"2024-03-04T16:14:37.387218+0000","last_clean":"2024-03-04T16:14:37.387218+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:14:37.387218+0000","last_undegraded":"2024-03-04T16:14:37.387218+0000","last_fullsized":"2024-03-04T16:14:37.387218+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:12:26.939557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.939557+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":524,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.382506+0000","last_change":"2024-03-04T16:12:03.608848+0000","last_active":"2024-03-04T16:14:37.382506+0000","last_peered":"2024-03-04T16:14:37.382506+0000","last_clean":"2024-03-04T16:14:37.382506+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:14:37.382506+0000","last_undegraded":"2024-03-04T16:14:37.382506+0000","last_fullsized":"2024-03-04T16:14:37.382506+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:03.608639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.608639+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":617,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389983+0000","last_change":"2024-03-04T16:13:02.118333+0000","last_active":"2024-03-04T16:14:37.389983+0000","last_peered":"2024-03-04T16:14:37.389983+0000","last_clean":"2024-03-04T16:14:37.389983+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:14:37.389983+0000","last_undegraded":"2024-03-04T16:14:37.389983+0000","last_fullsized":"2024-03-04T16:14:37.389983+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:13:02.118207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:02.118207+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"272'8468","reported_seq":8538,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.183446+0000","last_change":"2024-03-04T16:12:58.258448+0000","last_active":"2024-03-04T16:14:38.183446+0000","last_peered":"2024-03-04T16:14:38.183446+0000","last_clean":"2024-03-04T16:14:38.183446+0000","last_became_active":"2024-03-04T16:12:58.258106+0000","last_became_peered":"2024-03-04T16:12:58.258106+0000","last_unstale":"2024-03-04T16:14:38.183446+0000","last_undegraded":"2024-03-04T16:14:38.183446+0000","last_fullsized":"2024-03-04T16:14:38.183446+0000","mapping_epoch":242,"log_start":"263'4900","ondisk_log_start":"263'4900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3568,"log_dups_size":0,"ondisk_log_size":3568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554958848,"num_objects":8468,"num_object_clones":0,"num_object_copies":25404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8468,"num_write_kb":541952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":444,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385277+0000","last_change":"2024-03-04T16:13:52.548216+0000","last_active":"2024-03-04T16:14:37.385277+0000","last_peered":"2024-03-04T16:14:37.385277+0000","last_clean":"2024-03-04T16:14:37.385277+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:14:37.385277+0000","last_undegraded":"2024-03-04T16:14:37.385277+0000","last_fullsized":"2024-03-04T16:14:37.385277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.053562+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":464,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386679+0000","last_change":"2024-03-04T16:12:04.845679+0000","last_active":"2024-03-04T16:14:37.386679+0000","last_peered":"2024-03-04T16:14:37.386679+0000","last_clean":"2024-03-04T16:14:37.386679+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:14:37.386679+0000","last_undegraded":"2024-03-04T16:14:37.386679+0000","last_fullsized":"2024-03-04T16:14:37.386679+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.845559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.845559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":492,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397743+0000","last_change":"2024-03-04T16:13:19.438912+0000","last_active":"2024-03-04T16:14:37.397743+0000","last_peered":"2024-03-04T16:14:37.397743+0000","last_clean":"2024-03-04T16:14:37.397743+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:14:37.397743+0000","last_undegraded":"2024-03-04T16:14:37.397743+0000","last_fullsized":"2024-03-04T16:14:37.397743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:13:19.438774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.438774+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":362,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388657+0000","last_change":"2024-03-04T16:13:55.457911+0000","last_active":"2024-03-04T16:14:37.388657+0000","last_peered":"2024-03-04T16:14:37.388657+0000","last_clean":"2024-03-04T16:14:37.388657+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:14:37.388657+0000","last_undegraded":"2024-03-04T16:14:37.388657+0000","last_fullsized":"2024-03-04T16:14:37.388657+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.457585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.457585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:05.916401+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"272'8255","reported_seq":8323,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.712478+0000","last_change":"2024-03-04T16:12:58.259241+0000","last_active":"2024-03-04T16:14:37.712478+0000","last_peered":"2024-03-04T16:14:37.712478+0000","last_clean":"2024-03-04T16:14:37.712478+0000","last_became_active":"2024-03-04T16:12:58.258565+0000","last_became_peered":"2024-03-04T16:12:58.258565+0000","last_unstale":"2024-03-04T16:14:37.712478+0000","last_undegraded":"2024-03-04T16:14:37.712478+0000","last_fullsized":"2024-03-04T16:14:37.712478+0000","mapping_epoch":242,"log_start":"260'3900","ondisk_log_start":"260'3900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540999680,"num_objects":8255,"num_object_clones":0,"num_object_copies":24765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8255,"num_write_kb":528320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385851+0000","last_change":"2024-03-04T16:11:59.935508+0000","last_active":"2024-03-04T16:14:37.385851+0000","last_peered":"2024-03-04T16:14:37.385851+0000","last_clean":"2024-03-04T16:14:37.385851+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:14:37.385851+0000","last_undegraded":"2024-03-04T16:14:37.385851+0000","last_fullsized":"2024-03-04T16:14:37.385851+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.935397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.935397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":183,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390354+0000","last_change":"2024-03-04T16:13:19.665876+0000","last_active":"2024-03-04T16:14:37.390354+0000","last_peered":"2024-03-04T16:14:37.390354+0000","last_clean":"2024-03-04T16:14:37.390354+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:14:37.390354+0000","last_undegraded":"2024-03-04T16:14:37.390354+0000","last_fullsized":"2024-03-04T16:14:37.390354+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.665604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.665604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":470,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387061+0000","last_change":"2024-03-04T16:12:10.817851+0000","last_active":"2024-03-04T16:14:37.387061+0000","last_peered":"2024-03-04T16:14:37.387061+0000","last_clean":"2024-03-04T16:14:37.387061+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:14:37.387061+0000","last_undegraded":"2024-03-04T16:14:37.387061+0000","last_fullsized":"2024-03-04T16:14:37.387061+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:10.817671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:12:10.817671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":270,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389512+0000","last_change":"2024-03-04T16:13:13.647026+0000","last_active":"2024-03-04T16:14:37.389512+0000","last_peered":"2024-03-04T16:14:37.389512+0000","last_clean":"2024-03-04T16:14:37.389512+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:14:37.389512+0000","last_undegraded":"2024-03-04T16:14:37.389512+0000","last_fullsized":"2024-03-04T16:14:37.389512+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.646813+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636800+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"272'8378","reported_seq":8448,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.683713+0000","last_change":"2024-03-04T16:12:58.259225+0000","last_active":"2024-03-04T16:14:37.683713+0000","last_peered":"2024-03-04T16:14:37.683713+0000","last_clean":"2024-03-04T16:14:37.683713+0000","last_became_active":"2024-03-04T16:12:58.258787+0000","last_became_peered":"2024-03-04T16:12:58.258787+0000","last_unstale":"2024-03-04T16:14:37.683713+0000","last_undegraded":"2024-03-04T16:14:37.683713+0000","last_fullsized":"2024-03-04T16:14:37.683713+0000","mapping_epoch":242,"log_start":"260'4000","ondisk_log_start":"260'4000","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":549060608,"num_objects":8378,"num_object_clones":0,"num_object_copies":25134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8378,"num_write_kb":536192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":339,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389705+0000","last_change":"2024-03-04T16:13:53.931017+0000","last_active":"2024-03-04T16:14:37.389705+0000","last_peered":"2024-03-04T16:14:37.389705+0000","last_clean":"2024-03-04T16:14:37.389705+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:14:37.389705+0000","last_undegraded":"2024-03-04T16:14:37.389705+0000","last_fullsized":"2024-03-04T16:14:37.389705+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.930927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.930927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.204703+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":176,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397967+0000","last_change":"2024-03-04T16:10:17.463035+0000","last_active":"2024-03-04T16:14:37.397967+0000","last_peered":"2024-03-04T16:14:37.397967+0000","last_clean":"2024-03-04T16:14:37.397967+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:14:37.397967+0000","last_undegraded":"2024-03-04T16:14:37.397967+0000","last_fullsized":"2024-03-04T16:14:37.397967+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.943544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.943544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":469,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380385+0000","last_change":"2024-03-04T16:13:07.694160+0000","last_active":"2024-03-04T16:14:37.380385+0000","last_peered":"2024-03-04T16:14:37.380385+0000","last_clean":"2024-03-04T16:14:37.380385+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:14:37.380385+0000","last_undegraded":"2024-03-04T16:14:37.380385+0000","last_fullsized":"2024-03-04T16:14:37.380385+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:07.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:13:07.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":319,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390204+0000","last_change":"2024-03-04T16:13:16.698881+0000","last_active":"2024-03-04T16:14:37.390204+0000","last_peered":"2024-03-04T16:14:37.390204+0000","last_clean":"2024-03-04T16:14:37.390204+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:14:37.390204+0000","last_undegraded":"2024-03-04T16:14:37.390204+0000","last_fullsized":"2024-03-04T16:14:37.390204+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.698648+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:40.220710+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"272'8456","reported_seq":8526,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.279652+0000","last_change":"2024-03-04T16:12:58.259820+0000","last_active":"2024-03-04T16:14:38.279652+0000","last_peered":"2024-03-04T16:14:38.279652+0000","last_clean":"2024-03-04T16:14:38.279652+0000","last_became_active":"2024-03-04T16:12:58.259075+0000","last_became_peered":"2024-03-04T16:12:58.259075+0000","last_unstale":"2024-03-04T16:14:38.279652+0000","last_undegraded":"2024-03-04T16:14:38.279652+0000","last_fullsized":"2024-03-04T16:14:38.279652+0000","mapping_epoch":242,"log_start":"261'4300","ondisk_log_start":"261'4300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554172416,"num_objects":8456,"num_object_clones":0,"num_object_copies":25368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8456,"num_write_kb":541184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":436,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.382357+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:14:37.382357+0000","last_peered":"2024-03-04T16:14:37.382357+0000","last_clean":"2024-03-04T16:14:37.382357+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:14:37.382357+0000","last_undegraded":"2024-03-04T16:14:37.382357+0000","last_fullsized":"2024-03-04T16:14:37.382357+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766578+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":484,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380252+0000","last_change":"2024-03-04T16:12:58.666189+0000","last_active":"2024-03-04T16:14:37.380252+0000","last_peered":"2024-03-04T16:14:37.380252+0000","last_clean":"2024-03-04T16:14:37.380252+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:14:37.380252+0000","last_undegraded":"2024-03-04T16:14:37.380252+0000","last_fullsized":"2024-03-04T16:14:37.380252+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:12:58.666070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:12:58.666070+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":482,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398173+0000","last_change":"2024-03-04T16:12:25.567411+0000","last_active":"2024-03-04T16:14:37.398173+0000","last_peered":"2024-03-04T16:14:37.398173+0000","last_clean":"2024-03-04T16:14:37.398173+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:14:37.398173+0000","last_undegraded":"2024-03-04T16:14:37.398173+0000","last_fullsized":"2024-03-04T16:14:37.398173+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:25.567295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.567295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":717,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390586+0000","last_change":"2024-03-04T16:12:18.618110+0000","last_active":"2024-03-04T16:14:37.390586+0000","last_peered":"2024-03-04T16:14:37.390586+0000","last_clean":"2024-03-04T16:14:37.390586+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:14:37.390586+0000","last_undegraded":"2024-03-04T16:14:37.390586+0000","last_fullsized":"2024-03-04T16:14:37.390586+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'30","last_scrub_stamp":"2024-03-04T16:12:18.617922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.617922+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636435+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":300,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.393695+0000","last_change":"2024-03-04T16:13:52.478892+0000","last_active":"2024-03-04T16:14:37.393695+0000","last_peered":"2024-03-04T16:14:37.393695+0000","last_clean":"2024-03-04T16:14:37.393695+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:14:37.393695+0000","last_undegraded":"2024-03-04T16:14:37.393695+0000","last_fullsized":"2024-03-04T16:14:37.393695+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.478776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.478776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.051799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"272'8393","reported_seq":8463,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.706614+0000","last_change":"2024-03-04T16:12:58.263282+0000","last_active":"2024-03-04T16:14:37.706614+0000","last_peered":"2024-03-04T16:14:37.706614+0000","last_clean":"2024-03-04T16:14:37.706614+0000","last_became_active":"2024-03-04T16:12:58.262950+0000","last_became_peered":"2024-03-04T16:12:58.262950+0000","last_unstale":"2024-03-04T16:14:37.706614+0000","last_undegraded":"2024-03-04T16:14:37.706614+0000","last_fullsized":"2024-03-04T16:14:37.706614+0000","mapping_epoch":242,"log_start":"260'4000","ondisk_log_start":"260'4000","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":550043648,"num_objects":8393,"num_object_clones":0,"num_object_copies":25179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8393,"num_write_kb":537152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":458,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386952+0000","last_change":"2024-03-04T16:13:20.064259+0000","last_active":"2024-03-04T16:14:37.386952+0000","last_peered":"2024-03-04T16:14:37.386952+0000","last_clean":"2024-03-04T16:14:37.386952+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:14:37.386952+0000","last_undegraded":"2024-03-04T16:14:37.386952+0000","last_fullsized":"2024-03-04T16:14:37.386952+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:20.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:20.064148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":468,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.389873+0000","last_change":"2024-03-04T16:10:29.352376+0000","last_active":"2024-03-04T16:14:37.389873+0000","last_peered":"2024-03-04T16:14:37.389873+0000","last_clean":"2024-03-04T16:14:37.389873+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:14:37.389873+0000","last_undegraded":"2024-03-04T16:14:37.389873+0000","last_fullsized":"2024-03-04T16:14:37.389873+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.897896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.897896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636364+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.384410+0000","last_change":"2024-03-04T16:14:01.534096+0000","last_active":"2024-03-04T16:14:37.384410+0000","last_peered":"2024-03-04T16:14:37.384410+0000","last_clean":"2024-03-04T16:14:37.384410+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:14:37.384410+0000","last_undegraded":"2024-03-04T16:14:37.384410+0000","last_fullsized":"2024-03-04T16:14:37.384410+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.533961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.533961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:08.148103+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":99,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390852+0000","last_change":"2024-03-04T16:12:35.728287+0000","last_active":"2024-03-04T16:14:37.390852+0000","last_peered":"2024-03-04T16:14:37.390852+0000","last_clean":"2024-03-04T16:14:37.390852+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:14:37.390852+0000","last_undegraded":"2024-03-04T16:14:37.390852+0000","last_fullsized":"2024-03-04T16:14:37.390852+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'18","last_scrub_stamp":"2024-03-04T16:11:57.950523+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:11:57.950523+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":176,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381150+0000","last_change":"2024-03-04T16:10:17.466321+0000","last_active":"2024-03-04T16:14:37.381150+0000","last_peered":"2024-03-04T16:14:37.381150+0000","last_clean":"2024-03-04T16:14:37.381150+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:14:37.381150+0000","last_undegraded":"2024-03-04T16:14:37.381150+0000","last_fullsized":"2024-03-04T16:14:37.381150+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:00.988847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.988847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"272'8349","reported_seq":8419,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.584294+0000","last_change":"2024-03-04T16:12:58.256227+0000","last_active":"2024-03-04T16:14:38.584294+0000","last_peered":"2024-03-04T16:14:38.584294+0000","last_clean":"2024-03-04T16:14:38.584294+0000","last_became_active":"2024-03-04T16:12:58.255897+0000","last_became_peered":"2024-03-04T16:12:58.255897+0000","last_unstale":"2024-03-04T16:14:38.584294+0000","last_undegraded":"2024-03-04T16:14:38.584294+0000","last_fullsized":"2024-03-04T16:14:38.584294+0000","mapping_epoch":242,"log_start":"261'4300","ondisk_log_start":"261'4300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4049,"log_dups_size":0,"ondisk_log_size":4049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.775512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":547160064,"num_objects":8349,"num_object_clones":0,"num_object_copies":25047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8349,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":451,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381552+0000","last_change":"2024-03-04T16:12:23.816872+0000","last_active":"2024-03-04T16:14:37.381552+0000","last_peered":"2024-03-04T16:14:37.381552+0000","last_clean":"2024-03-04T16:14:37.381552+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:14:37.381552+0000","last_undegraded":"2024-03-04T16:14:37.381552+0000","last_fullsized":"2024-03-04T16:14:37.381552+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.816724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.816724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":771,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386019+0000","last_change":"2024-03-04T16:12:59.102019+0000","last_active":"2024-03-04T16:14:37.386019+0000","last_peered":"2024-03-04T16:14:37.386019+0000","last_clean":"2024-03-04T16:14:37.386019+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:14:37.386019+0000","last_undegraded":"2024-03-04T16:14:37.386019+0000","last_fullsized":"2024-03-04T16:14:37.386019+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:12:59.101913+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:12:59.101913+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381905+0000","last_change":"2024-03-04T16:13:56.375663+0000","last_active":"2024-03-04T16:14:37.381905+0000","last_peered":"2024-03-04T16:14:37.381905+0000","last_clean":"2024-03-04T16:14:37.381905+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:14:37.381905+0000","last_undegraded":"2024-03-04T16:14:37.381905+0000","last_fullsized":"2024-03-04T16:14:37.381905+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.375405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.375405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.685421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":296,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399449+0000","last_change":"2024-03-04T16:10:29.352761+0000","last_active":"2024-03-04T16:14:37.399449+0000","last_peered":"2024-03-04T16:14:37.399449+0000","last_clean":"2024-03-04T16:14:37.399449+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:14:37.399449+0000","last_undegraded":"2024-03-04T16:14:37.399449+0000","last_fullsized":"2024-03-04T16:14:37.399449+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'16","last_scrub_stamp":"2024-03-04T16:10:00.911711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:00.911711+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386604+0000","last_change":"2024-03-04T16:12:09.805823+0000","last_active":"2024-03-04T16:14:37.386604+0000","last_peered":"2024-03-04T16:14:37.386604+0000","last_clean":"2024-03-04T16:14:37.386604+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:14:37.386604+0000","last_undegraded":"2024-03-04T16:14:37.386604+0000","last_fullsized":"2024-03-04T16:14:37.386604+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:09.805715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:09.805715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"272'8308","reported_seq":8378,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.286350+0000","last_change":"2024-03-04T16:12:58.258704+0000","last_active":"2024-03-04T16:14:38.286350+0000","last_peered":"2024-03-04T16:14:38.286350+0000","last_clean":"2024-03-04T16:14:38.286350+0000","last_became_active":"2024-03-04T16:12:58.258386+0000","last_became_peered":"2024-03-04T16:12:58.258386+0000","last_unstale":"2024-03-04T16:14:38.286350+0000","last_undegraded":"2024-03-04T16:14:38.286350+0000","last_fullsized":"2024-03-04T16:14:38.286350+0000","mapping_epoch":242,"log_start":"261'4100","ondisk_log_start":"261'4100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":544473088,"num_objects":8308,"num_object_clones":0,"num_object_copies":24924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":531712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":452,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390182+0000","last_change":"2024-03-04T16:12:26.602465+0000","last_active":"2024-03-04T16:14:37.390182+0000","last_peered":"2024-03-04T16:14:37.390182+0000","last_clean":"2024-03-04T16:14:37.390182+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:14:37.390182+0000","last_undegraded":"2024-03-04T16:14:37.390182+0000","last_fullsized":"2024-03-04T16:14:37.390182+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:26.602350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.602350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636562+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":308,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390685+0000","last_change":"2024-03-04T16:13:50.496078+0000","last_active":"2024-03-04T16:14:37.390685+0000","last_peered":"2024-03-04T16:14:37.390685+0000","last_clean":"2024-03-04T16:14:37.390685+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:14:37.390685+0000","last_undegraded":"2024-03-04T16:14:37.390685+0000","last_fullsized":"2024-03-04T16:14:37.390685+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:50.495814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:50.495814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.556381+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1192,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385854+0000","last_change":"2024-03-04T16:13:57.825817+0000","last_active":"2024-03-04T16:14:37.385854+0000","last_peered":"2024-03-04T16:14:37.385854+0000","last_clean":"2024-03-04T16:14:37.385854+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:14:37.385854+0000","last_undegraded":"2024-03-04T16:14:37.385854+0000","last_fullsized":"2024-03-04T16:14:37.385854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:13:57.825680+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:13:57.825680+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.150143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":707,"num_read_kb":702,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":352,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385859+0000","last_change":"2024-03-04T16:13:15.734217+0000","last_active":"2024-03-04T16:14:37.385859+0000","last_peered":"2024-03-04T16:14:37.385859+0000","last_clean":"2024-03-04T16:14:37.385859+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:14:37.385859+0000","last_undegraded":"2024-03-04T16:14:37.385859+0000","last_fullsized":"2024-03-04T16:14:37.385859+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:13:15.734074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.734074+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:45.507334+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":463,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381928+0000","last_change":"2024-03-04T16:14:06.002348+0000","last_active":"2024-03-04T16:14:37.381928+0000","last_peered":"2024-03-04T16:14:37.381928+0000","last_clean":"2024-03-04T16:14:37.381928+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:14:37.381928+0000","last_undegraded":"2024-03-04T16:14:37.381928+0000","last_fullsized":"2024-03-04T16:14:37.381928+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:06.002229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.002229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:30.238482+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"272'8323","reported_seq":8393,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.462790+0000","last_change":"2024-03-04T16:12:58.261720+0000","last_active":"2024-03-04T16:14:37.462790+0000","last_peered":"2024-03-04T16:14:37.462790+0000","last_clean":"2024-03-04T16:14:37.462790+0000","last_became_active":"2024-03-04T16:12:58.261282+0000","last_became_peered":"2024-03-04T16:12:58.261282+0000","last_unstale":"2024-03-04T16:14:37.462790+0000","last_undegraded":"2024-03-04T16:14:37.462790+0000","last_fullsized":"2024-03-04T16:14:37.462790+0000","mapping_epoch":242,"log_start":"261'4200","ondisk_log_start":"261'4200","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4123,"log_dups_size":0,"ondisk_log_size":4123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":545456128,"num_objects":8323,"num_object_clones":0,"num_object_copies":24969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":532672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":464,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.392752+0000","last_change":"2024-03-04T16:13:59.616838+0000","last_active":"2024-03-04T16:14:37.392752+0000","last_peered":"2024-03-04T16:14:37.392752+0000","last_clean":"2024-03-04T16:14:37.392752+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:14:37.392752+0000","last_undegraded":"2024-03-04T16:14:37.392752+0000","last_fullsized":"2024-03-04T16:14:37.392752+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.616709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.616709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.656637+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390435+0000","last_change":"2024-03-04T16:13:05.550936+0000","last_active":"2024-03-04T16:14:37.390435+0000","last_peered":"2024-03-04T16:14:37.390435+0000","last_clean":"2024-03-04T16:14:37.390435+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:14:37.390435+0000","last_undegraded":"2024-03-04T16:14:37.390435+0000","last_fullsized":"2024-03-04T16:14:37.390435+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.550829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.550829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":438,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380191+0000","last_change":"2024-03-04T16:12:17.635817+0000","last_active":"2024-03-04T16:14:37.380191+0000","last_peered":"2024-03-04T16:14:37.380191+0000","last_clean":"2024-03-04T16:14:37.380191+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:14:37.380191+0000","last_undegraded":"2024-03-04T16:14:37.380191+0000","last_fullsized":"2024-03-04T16:14:37.380191+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:12:17.635712+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:12:17.635712+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":167,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380375+0000","last_change":"2024-03-04T16:10:29.330444+0000","last_active":"2024-03-04T16:14:37.380375+0000","last_peered":"2024-03-04T16:14:37.380375+0000","last_clean":"2024-03-04T16:14:37.380375+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:14:37.380375+0000","last_undegraded":"2024-03-04T16:14:37.380375+0000","last_fullsized":"2024-03-04T16:14:37.380375+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:55.853684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:55.853684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"272'8472","reported_seq":8542,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.684767+0000","last_change":"2024-03-04T16:12:58.262941+0000","last_active":"2024-03-04T16:14:37.684767+0000","last_peered":"2024-03-04T16:14:37.684767+0000","last_clean":"2024-03-04T16:14:37.684767+0000","last_became_active":"2024-03-04T16:12:58.262611+0000","last_became_peered":"2024-03-04T16:12:58.262611+0000","last_unstale":"2024-03-04T16:14:37.684767+0000","last_undegraded":"2024-03-04T16:14:37.684767+0000","last_fullsized":"2024-03-04T16:14:37.684767+0000","mapping_epoch":242,"log_start":"260'4100","ondisk_log_start":"260'4100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":555220992,"num_objects":8472,"num_object_clones":0,"num_object_copies":25416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8472,"num_write_kb":542208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":466,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388900+0000","last_change":"2024-03-04T16:14:01.478625+0000","last_active":"2024-03-04T16:14:37.388900+0000","last_peered":"2024-03-04T16:14:37.388900+0000","last_clean":"2024-03-04T16:14:37.388900+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:14:37.388900+0000","last_undegraded":"2024-03-04T16:14:37.388900+0000","last_fullsized":"2024-03-04T16:14:37.388900+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.477978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.477978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:23.643549+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":475,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380959+0000","last_change":"2024-03-04T16:13:16.814942+0000","last_active":"2024-03-04T16:14:37.380959+0000","last_peered":"2024-03-04T16:14:37.380959+0000","last_clean":"2024-03-04T16:14:37.380959+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:14:37.380959+0000","last_undegraded":"2024-03-04T16:14:37.380959+0000","last_fullsized":"2024-03-04T16:14:37.380959+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.814825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.814825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:46.355001+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":495,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.394651+0000","last_change":"2024-03-04T16:13:16.606983+0000","last_active":"2024-03-04T16:14:37.394651+0000","last_peered":"2024-03-04T16:14:37.394651+0000","last_clean":"2024-03-04T16:14:37.394651+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:14:37.394651+0000","last_undegraded":"2024-03-04T16:14:37.394651+0000","last_fullsized":"2024-03-04T16:14:37.394651+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.605307+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636735+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":254,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.380434+0000","last_change":"2024-03-04T16:13:58.080312+0000","last_active":"2024-03-04T16:14:37.380434+0000","last_peered":"2024-03-04T16:14:37.380434+0000","last_clean":"2024-03-04T16:14:37.380434+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:14:37.380434+0000","last_undegraded":"2024-03-04T16:14:37.380434+0000","last_fullsized":"2024-03-04T16:14:37.380434+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:13:58.080163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.080163+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:19.165616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"272'8582","reported_seq":8652,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.590223+0000","last_change":"2024-03-04T16:12:58.259682+0000","last_active":"2024-03-04T16:14:38.590223+0000","last_peered":"2024-03-04T16:14:38.590223+0000","last_clean":"2024-03-04T16:14:38.590223+0000","last_became_active":"2024-03-04T16:12:58.259080+0000","last_became_peered":"2024-03-04T16:12:58.259080+0000","last_unstale":"2024-03-04T16:14:38.590223+0000","last_undegraded":"2024-03-04T16:14:38.590223+0000","last_fullsized":"2024-03-04T16:14:38.590223+0000","mapping_epoch":242,"log_start":"261'4500","ondisk_log_start":"261'4500","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.775408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":562429952,"num_objects":8582,"num_object_clones":0,"num_object_copies":25746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8582,"num_write_kb":549248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":464,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386558+0000","last_change":"2024-03-04T16:11:59.019162+0000","last_active":"2024-03-04T16:14:37.386558+0000","last_peered":"2024-03-04T16:14:37.386558+0000","last_clean":"2024-03-04T16:14:37.386558+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:14:37.386558+0000","last_undegraded":"2024-03-04T16:14:37.386558+0000","last_fullsized":"2024-03-04T16:14:37.386558+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.019082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.019082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":312,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.613319+0000","last_change":"2024-03-04T16:14:37.613350+0000","last_active":"2024-03-04T16:14:37.613319+0000","last_peered":"2024-03-04T16:14:37.613319+0000","last_clean":"2024-03-04T16:14:37.613319+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:14:37.613319+0000","last_undegraded":"2024-03-04T16:14:37.613319+0000","last_fullsized":"2024-03-04T16:14:37.613319+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:14:37.613237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:37.613237+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:51.805994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":523,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397592+0000","last_change":"2024-03-04T16:12:24.566421+0000","last_active":"2024-03-04T16:14:37.397592+0000","last_peered":"2024-03-04T16:14:37.397592+0000","last_clean":"2024-03-04T16:14:37.397592+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:14:37.397592+0000","last_undegraded":"2024-03-04T16:14:37.397592+0000","last_fullsized":"2024-03-04T16:14:37.397592+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:12:24.566235+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.566235+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":591,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.386121+0000","last_change":"2024-03-04T16:09:59.969245+0000","last_active":"2024-03-04T16:14:37.386121+0000","last_peered":"2024-03-04T16:14:37.386121+0000","last_clean":"2024-03-04T16:14:37.386121+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:14:37.386121+0000","last_undegraded":"2024-03-04T16:14:37.386121+0000","last_fullsized":"2024-03-04T16:14:37.386121+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:09:59.969166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.969166+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"272'8224","reported_seq":8294,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.456545+0000","last_change":"2024-03-04T16:12:58.261655+0000","last_active":"2024-03-04T16:14:37.456545+0000","last_peered":"2024-03-04T16:14:37.456545+0000","last_clean":"2024-03-04T16:14:37.456545+0000","last_became_active":"2024-03-04T16:12:58.261321+0000","last_became_peered":"2024-03-04T16:12:58.261321+0000","last_unstale":"2024-03-04T16:14:37.456545+0000","last_undegraded":"2024-03-04T16:14:37.456545+0000","last_fullsized":"2024-03-04T16:14:37.456545+0000","mapping_epoch":242,"log_start":"261'4100","ondisk_log_start":"261'4100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4124,"log_dups_size":0,"ondisk_log_size":4124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":538968064,"num_objects":8224,"num_object_clones":0,"num_object_copies":24672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8224,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8224,"num_write_kb":526336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397576+0000","last_change":"2024-03-04T16:13:15.438272+0000","last_active":"2024-03-04T16:14:37.397576+0000","last_peered":"2024-03-04T16:14:37.397576+0000","last_clean":"2024-03-04T16:14:37.397576+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:14:37.397576+0000","last_undegraded":"2024-03-04T16:14:37.397576+0000","last_fullsized":"2024-03-04T16:14:37.397576+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:15.438148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.438148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":460,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387504+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:14:37.387504+0000","last_peered":"2024-03-04T16:14:37.387504+0000","last_clean":"2024-03-04T16:14:37.387504+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:14:37.387504+0000","last_undegraded":"2024-03-04T16:14:37.387504+0000","last_fullsized":"2024-03-04T16:14:37.387504+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766604+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":550,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385841+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:14:37.385841+0000","last_peered":"2024-03-04T16:14:37.385841+0000","last_clean":"2024-03-04T16:14:37.385841+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:14:37.385841+0000","last_undegraded":"2024-03-04T16:14:37.385841+0000","last_fullsized":"2024-03-04T16:14:37.385841+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766746+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":448,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397306+0000","last_change":"2024-03-04T16:10:01.801431+0000","last_active":"2024-03-04T16:14:37.397306+0000","last_peered":"2024-03-04T16:14:37.397306+0000","last_clean":"2024-03-04T16:14:37.397306+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:14:37.397306+0000","last_undegraded":"2024-03-04T16:14:37.397306+0000","last_fullsized":"2024-03-04T16:14:37.397306+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.801308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.801308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"272'8332","reported_seq":8402,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.593277+0000","last_change":"2024-03-04T16:12:58.259473+0000","last_active":"2024-03-04T16:14:38.593277+0000","last_peered":"2024-03-04T16:14:38.593277+0000","last_clean":"2024-03-04T16:14:38.593277+0000","last_became_active":"2024-03-04T16:12:58.259164+0000","last_became_peered":"2024-03-04T16:12:58.259164+0000","last_unstale":"2024-03-04T16:14:38.593277+0000","last_undegraded":"2024-03-04T16:14:38.593277+0000","last_fullsized":"2024-03-04T16:14:38.593277+0000","mapping_epoch":242,"log_start":"261'4300","ondisk_log_start":"261'4300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.775443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":546045952,"num_objects":8332,"num_object_clones":0,"num_object_copies":24996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8332,"num_write_kb":533248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":455,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381099+0000","last_change":"2024-03-04T16:10:04.246434+0000","last_active":"2024-03-04T16:14:37.381099+0000","last_peered":"2024-03-04T16:14:37.381099+0000","last_clean":"2024-03-04T16:14:37.381099+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:14:37.381099+0000","last_undegraded":"2024-03-04T16:14:37.381099+0000","last_fullsized":"2024-03-04T16:14:37.381099+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:04.246203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:04.246203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":599,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399647+0000","last_change":"2024-03-04T16:11:59.819002+0000","last_active":"2024-03-04T16:14:37.399647+0000","last_peered":"2024-03-04T16:14:37.399647+0000","last_clean":"2024-03-04T16:14:37.399647+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:14:37.399647+0000","last_undegraded":"2024-03-04T16:14:37.399647+0000","last_fullsized":"2024-03-04T16:14:37.399647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:11:59.818881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.818881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.385744+0000","last_change":"2024-03-04T16:13:58.414151+0000","last_active":"2024-03-04T16:14:37.385744+0000","last_peered":"2024-03-04T16:14:37.385744+0000","last_clean":"2024-03-04T16:14:37.385744+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:14:37.385744+0000","last_undegraded":"2024-03-04T16:14:37.385744+0000","last_fullsized":"2024-03-04T16:14:37.385744+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.413756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.413756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:09.771485+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"240'27","reported_seq":495,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.391330+0000","last_change":"2024-03-04T16:12:24.577754+0000","last_active":"2024-03-04T16:14:37.391330+0000","last_peered":"2024-03-04T16:14:37.391330+0000","last_clean":"2024-03-04T16:14:37.391330+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:14:37.391330+0000","last_undegraded":"2024-03-04T16:14:37.391330+0000","last_fullsized":"2024-03-04T16:14:37.391330+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:24.577573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.577573+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636533+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"272'8181","reported_seq":8251,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.466000+0000","last_change":"2024-03-04T16:12:58.262002+0000","last_active":"2024-03-04T16:14:37.466000+0000","last_peered":"2024-03-04T16:14:37.466000+0000","last_clean":"2024-03-04T16:14:37.466000+0000","last_became_active":"2024-03-04T16:12:58.261597+0000","last_became_peered":"2024-03-04T16:12:58.261597+0000","last_unstale":"2024-03-04T16:14:37.466000+0000","last_undegraded":"2024-03-04T16:14:37.466000+0000","last_fullsized":"2024-03-04T16:14:37.466000+0000","mapping_epoch":242,"log_start":"261'4100","ondisk_log_start":"261'4100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4081,"log_dups_size":0,"ondisk_log_size":4081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:37.985332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":536150016,"num_objects":8181,"num_object_clones":0,"num_object_copies":24543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8181,"num_write_kb":523584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":447,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.388288+0000","last_change":"2024-03-04T16:12:26.381739+0000","last_active":"2024-03-04T16:14:37.388288+0000","last_peered":"2024-03-04T16:14:37.388288+0000","last_clean":"2024-03-04T16:14:37.388288+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:14:37.388288+0000","last_undegraded":"2024-03-04T16:14:37.388288+0000","last_fullsized":"2024-03-04T16:14:37.388288+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:12:26.381198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.381198+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766868+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":461,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.381857+0000","last_change":"2024-03-04T16:13:17.814252+0000","last_active":"2024-03-04T16:14:37.381857+0000","last_peered":"2024-03-04T16:14:37.381857+0000","last_clean":"2024-03-04T16:14:37.381857+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:14:37.381857+0000","last_undegraded":"2024-03-04T16:14:37.381857+0000","last_fullsized":"2024-03-04T16:14:37.381857+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.814150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.814150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:39.197394+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":462,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397395+0000","last_change":"2024-03-04T16:13:13.359950+0000","last_active":"2024-03-04T16:14:37.397395+0000","last_peered":"2024-03-04T16:14:37.397395+0000","last_clean":"2024-03-04T16:14:37.397395+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:14:37.397395+0000","last_undegraded":"2024-03-04T16:14:37.397395+0000","last_fullsized":"2024-03-04T16:14:37.397395+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.359809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.359809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:39.651630+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":514,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399431+0000","last_change":"2024-03-04T16:14:06.377160+0000","last_active":"2024-03-04T16:14:37.399431+0000","last_peered":"2024-03-04T16:14:37.399431+0000","last_clean":"2024-03-04T16:14:37.399431+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:14:37.399431+0000","last_undegraded":"2024-03-04T16:14:37.399431+0000","last_fullsized":"2024-03-04T16:14:37.399431+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:14:06.377028+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.377028+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.277935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"272'8280","reported_seq":8350,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.178815+0000","last_change":"2024-03-04T16:12:58.261338+0000","last_active":"2024-03-04T16:14:38.178815+0000","last_peered":"2024-03-04T16:14:38.178815+0000","last_clean":"2024-03-04T16:14:38.178815+0000","last_became_active":"2024-03-04T16:12:58.260993+0000","last_became_peered":"2024-03-04T16:12:58.260993+0000","last_unstale":"2024-03-04T16:14:38.178815+0000","last_undegraded":"2024-03-04T16:14:38.178815+0000","last_fullsized":"2024-03-04T16:14:38.178815+0000","mapping_epoch":242,"log_start":"262'4700","ondisk_log_start":"262'4700","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3580,"log_dups_size":0,"ondisk_log_size":3580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.636698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542638080,"num_objects":8280,"num_object_clones":0,"num_object_copies":24840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8280,"num_write_kb":529920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":189,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387353+0000","last_change":"2024-03-04T16:13:55.518681+0000","last_active":"2024-03-04T16:14:37.387353+0000","last_peered":"2024-03-04T16:14:37.387353+0000","last_clean":"2024-03-04T16:14:37.387353+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:14:37.387353+0000","last_undegraded":"2024-03-04T16:14:37.387353+0000","last_fullsized":"2024-03-04T16:14:37.387353+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:14.056169+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":456,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.390685+0000","last_change":"2024-03-04T16:13:11.618623+0000","last_active":"2024-03-04T16:14:37.390685+0000","last_peered":"2024-03-04T16:14:37.390685+0000","last_clean":"2024-03-04T16:14:37.390685+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:14:37.390685+0000","last_undegraded":"2024-03-04T16:14:37.390685+0000","last_fullsized":"2024-03-04T16:14:37.390685+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:11.618510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:11.618510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.397662+0000","last_change":"2024-03-04T16:13:16.430026+0000","last_active":"2024-03-04T16:14:37.397662+0000","last_peered":"2024-03-04T16:14:37.397662+0000","last_clean":"2024-03-04T16:14:37.397662+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:14:37.397662+0000","last_undegraded":"2024-03-04T16:14:37.397662+0000","last_fullsized":"2024-03-04T16:14:37.397662+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.429889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.429889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":557,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399279+0000","last_change":"2024-03-04T16:13:54.369721+0000","last_active":"2024-03-04T16:14:37.399279+0000","last_peered":"2024-03-04T16:14:37.399279+0000","last_clean":"2024-03-04T16:14:37.399279+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:14:37.399279+0000","last_undegraded":"2024-03-04T16:14:37.399279+0000","last_fullsized":"2024-03-04T16:14:37.399279+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:13:54.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.369577+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.639737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"272'8433","reported_seq":8503,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:38.562882+0000","last_change":"2024-03-04T16:12:58.259516+0000","last_active":"2024-03-04T16:14:38.562882+0000","last_peered":"2024-03-04T16:14:38.562882+0000","last_clean":"2024-03-04T16:14:38.562882+0000","last_became_active":"2024-03-04T16:12:58.259119+0000","last_became_peered":"2024-03-04T16:12:58.259119+0000","last_unstale":"2024-03-04T16:14:38.562882+0000","last_undegraded":"2024-03-04T16:14:38.562882+0000","last_fullsized":"2024-03-04T16:14:38.562882+0000","mapping_epoch":242,"log_start":"261'4400","ondisk_log_start":"261'4400","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4033,"log_dups_size":0,"ondisk_log_size":4033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.775361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":552665088,"num_objects":8433,"num_object_clones":0,"num_object_copies":25299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8433,"num_write_kb":539712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":450,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.399685+0000","last_change":"2024-03-04T16:12:00.783064+0000","last_active":"2024-03-04T16:14:37.399685+0000","last_peered":"2024-03-04T16:14:37.399685+0000","last_clean":"2024-03-04T16:14:37.399685+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:14:37.399685+0000","last_undegraded":"2024-03-04T16:14:37.399685+0000","last_fullsized":"2024-03-04T16:14:37.399685+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.782828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.782828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":327,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.387688+0000","last_change":"2024-03-04T16:13:18.019746+0000","last_active":"2024-03-04T16:14:37.387688+0000","last_peered":"2024-03-04T16:14:37.387688+0000","last_clean":"2024-03-04T16:14:37.387688+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:14:37.387688+0000","last_undegraded":"2024-03-04T16:14:37.387688+0000","last_fullsized":"2024-03-04T16:14:37.387688+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:13:18.019623+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.019623+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":460,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.384991+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:14:37.384991+0000","last_peered":"2024-03-04T16:14:37.384991+0000","last_clean":"2024-03-04T16:14:37.384991+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:14:37.384991+0000","last_undegraded":"2024-03-04T16:14:37.384991+0000","last_fullsized":"2024-03-04T16:14:37.384991+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:38.766686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":500,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-03-04T16:14:37.398906+0000","last_change":"2024-03-04T16:13:05.484192+0000","last_active":"2024-03-04T16:14:37.398906+0000","last_peered":"2024-03-04T16:14:37.398906+0000","last_clean":"2024-03-04T16:14:37.398906+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:14:37.398906+0000","last_undegraded":"2024-03-04T16:14:37.398906+0000","last_fullsized":"2024-03-04T16:14:37.398906+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:13:05.484009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.484009+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8776581120,"num_objects":133920,"num_object_clones":0,"num_object_copies":401760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":133920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":133920,"num_write_kb":8570880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":26306543616,"data_stored":26306543616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65720,"ondisk_log_size":65720,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":707,"num_read_kb":702,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070611,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5716604,"kb_used_data":5306236,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":88028548,"statfs":{"total":95995035648,"available":90141233152,"internally_reserved":0,"allocated":5433585664,"data_stored":5433155262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364951,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575025,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5869660,"kb_used_data":5318980,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":87875492,"statfs":{"total":95995035648,"available":89984503808,"internally_reserved":0,"allocated":5446635520,"data_stored":5446219977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3605204,"kb_used_data":3198908,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90139948,"statfs":{"total":95995035648,"available":92303306752,"internally_reserved":0,"allocated":3275681792,"data_stored":3275275336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411499,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149862,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4942744,"kb_used_data":4275640,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":88802408,"statfs":{"total":95995035648,"available":90933665792,"internally_reserved":0,"allocated":4378255360,"data_stored":4377852145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":683014600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953633,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2519612,"kb_used_data":2149412,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":91225540,"statfs":{"total":95995035648,"available":93414952960,"internally_reserved":0,"allocated":2200997888,"data_stored":2200613897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626589,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5849252,"kb_used_data":5304228,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":87895900,"statfs":{"total":95995035648,"available":90005401600,"internally_reserved":0,"allocated":5431529472,"data_stored":5431121295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3295412224,"data_stored":3295412224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5475729408,"data_stored":5475729408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4412014592,"data_stored":4412014592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2201681920,"data_stored":2201681920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5463539712,"data_stored":5463539712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5458165760,"data_stored":5458165760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:14:41.836 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-04T16:14:41.836 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:14:41.836 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-04T16:14:41.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:14:41.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:41 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:14:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:41 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:14:41.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:41.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:42.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:42.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:42.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:42.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.351 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:42.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:42.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:42.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:42.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:42.614 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:42.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: osdmap e273: 8 total, 8 up, 6 in 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: pgmap v536: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 828 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:14:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: osdmap e273: 8 total, 8 up, 6 in 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: pgmap v536: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 828 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:14:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:42 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:14:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.753 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:42.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: osdmap e273: 8 total, 8 up, 6 in 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: pgmap v536: 148 pgs: 148 active+clean; 8.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 828 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:14:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:14:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:42.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:42.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:42.993 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:43.016 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:43.119 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:43.154 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:43.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:43.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:43.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:43.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:43.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:43.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:43.521 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:43.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:43.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: 9.b scrub starts 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:14:43.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: 9.b scrub starts 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:14:43.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:43.745 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:43.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: 9.b scrub starts 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:14:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:43.879 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:43.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:43.981 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.015 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:44.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:44.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:44.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:44.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:44.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:44.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:44.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:44.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:44.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:44.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:44.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 79 MiB/s wr, 1.43k op/s 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[29779]: 9.9 scrub starts 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:14:44.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:14:44.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:14:44.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 79 MiB/s wr, 1.43k op/s 2024-03-04T16:14:44.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:44 smithi038 ceph-mon[38113]: 9.9 scrub starts 2024-03-04T16:14:44.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:44.856 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:44.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: pgmap v537: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 79 MiB/s wr, 1.43k op/s 2024-03-04T16:14:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:44 smithi176 ceph-mon[33619]: 9.9 scrub starts 2024-03-04T16:14:44.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:45.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:45.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:45.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:45.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.367 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:45.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:45.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.493 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:45.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:45.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.626 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:45.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:45.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:14:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:14:45.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:45.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:45.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:14:45.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:45 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:14:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:45.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:45 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:45 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:45 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:14:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:45 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:14:45.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:45.902 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:45.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:46.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:46.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:46.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:46.171 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:46.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:46.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:46.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:14:46.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:46.441 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:46.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:46.575 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: osdmap e274: 8 total, 8 up, 6 in 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: pgmap v539: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.63k op/s 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:14:46.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[29779]: 9.b scrub ok 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: osdmap e274: 8 total, 8 up, 6 in 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: pgmap v539: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.63k op/s 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:14:46.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:14:46.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:14:46.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:14:46.617 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:46 smithi038 ceph-mon[38113]: 9.b scrub ok 2024-03-04T16:14:46.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:46.705 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:46.722 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:46.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: osdmap e274: 8 total, 8 up, 6 in 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: pgmap v539: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 86 MiB/s wr, 1.63k op/s 2024-03-04T16:14:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:14:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:14:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:14:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:14:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:46 smithi176 ceph-mon[33619]: 9.b scrub ok 2024-03-04T16:14:46.944 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:46.974 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:46.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:47.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.108 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:47.209 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.244 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:47.345 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:47.373 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:47.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:47.509 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:47.551 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:14:47.552 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:14:47.553 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:47 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:14:47.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:47.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:47.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:47.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.780 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:47.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:14:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:14:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:14:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:14:47.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:14:47.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:47 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:14:47.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:47.922 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:47.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:47.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:48.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:48.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:48.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:48.199 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:48.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:48.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:48.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:48.470 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:48.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:48.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:48.594 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:48.619 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:48.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:48.729 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: 9.9 scrub ok 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: pgmap v540: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 36 MiB/s wr, 818 op/s 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:14:48.730 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: 9.9 scrub ok 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: pgmap v540: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 36 MiB/s wr, 818 op/s 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:14:48.731 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:48 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:14:48.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:48.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:48.773 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:14:48.814 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:14:48.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: 9.9 scrub ok 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: pgmap v540: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 8.3 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 767 B/s rd, 36 MiB/s wr, 818 op/s 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:14:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:48 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:14:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:48.876 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:48.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:48.887 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:14:48.928 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:14:48.969 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:14:48.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:49.008 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:14:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.013 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:49.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:49.055 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:14:49.059 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:14:49.060 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:14:49.060 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:14:49.060 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:14:49.060 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:14:49.060 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:14:49.061 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:14:49.061 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:14:49.114 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:49.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.429 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:49.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:49.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:49.567 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:49.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:14:49.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:14:49.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:14:49.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:14:49.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1709620760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:49.666 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:14:49.666 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:14:49.666 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:14:49.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:14:49.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1709620760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:49.669 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:49.701 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:49.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:49.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:49.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:49 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:14:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:49 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:14:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:49 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:14:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:49 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:14:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1709620760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:14:49.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:49.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:49.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:50.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.122 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:50.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:50.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:50.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:50.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:50.393 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:50.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:50.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.532 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:50.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:50.555 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:50.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.671 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:50.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:50.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:50.796 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[29779]: pgmap v541: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.59k op/s 2024-03-04T16:14:50.796 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:14:50.796 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:14:50.797 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[38113]: pgmap v541: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.59k op/s 2024-03-04T16:14:50.797 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:14:50.797 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:50 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:14:50.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:50.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:50 smithi176 ceph-mon[33619]: pgmap v541: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.59k op/s 2024-03-04T16:14:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:50 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:14:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:50 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:14:50.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:50.943 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:50.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:51.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:51.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:51.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:51.220 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:51.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:51.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:51.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:51.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:51.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:51.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:51.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:51.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:51.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:14:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:14:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:14:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: osdmap e275: 8 total, 8 up, 6 in 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:14:51.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:51 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:14:51.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:51.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:51.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: osdmap e275: 8 total, 8 up, 6 in 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:14:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: osdmap e275: 8 total, 8 up, 6 in 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:14:51.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:51 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:14:52.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:52.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.043 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:52.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:52.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.179 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:52.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:52.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:52.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:52.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:52.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:52.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:52.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:52.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.730 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:52.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:52.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: pgmap v543: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 7.5 MiB/s wr, 990 op/s 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:14:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: pgmap v543: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 7.5 MiB/s wr, 990 op/s 2024-03-04T16:14:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:14:52.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:52 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:14:52.831 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: pgmap v543: 148 pgs: 148 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 7.5 MiB/s wr, 990 op/s 2024-03-04T16:14:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:14:52.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:14:52.861 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:52.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:52.963 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:53.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.156 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:53.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:53.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:53.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:53.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:14:52 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:14:53.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.435 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:53.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:53.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:53.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:53.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: osdmap e276: 8 total, 8 up, 6 in 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:14:53.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: osdmap e276: 8 total, 8 up, 6 in 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:14:53.677 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:53.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:53.678 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:14:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:14:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:14:53.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:53.748 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:53.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: osdmap e276: 8 total, 8 up, 6 in 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:14:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:14:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:14:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:14:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:14:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:53.898 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:53.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:53.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:54.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:54.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:54.175 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:54.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:54.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:54.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:54.429 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:54.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:54.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:54.712 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:14:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:14:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: pgmap v545: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:14:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: pgmap v545: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:54.747 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:54 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:14:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.748 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:54.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:14:54.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: pgmap v545: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.6 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:14:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:54 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:14:54.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:54.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:54.983 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:55.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:55.043 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:14:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:14:55.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:55.150 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:55.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:55.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:55.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.428 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:55.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:55.529 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:55.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:55.627 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:14:55.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:55.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:55.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.824 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:55.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:55.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:55.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: 9.5 scrub starts 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[29779]: from='client.35441 v1:172.21.15.38:0/781887596' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:55.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: 9.5 scrub starts 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:14:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:55 smithi038 ceph-mon[38113]: from='client.35441 v1:172.21.15.38:0/781887596' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:56.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:56.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: 9.5 scrub starts 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:14:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:55 smithi176 ceph-mon[33619]: from='client.35441 v1:172.21.15.38:0/781887596' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:14:56.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:56.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:56.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:56.384 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:56.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:56.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:56.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:56.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.849 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:56.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:56.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:56.987 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:56.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:56.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: pgmap v546: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: osdmap e277: 8 total, 8 up, 6 in 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: pgmap v546: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:14:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:14:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: osdmap e277: 8 total, 8 up, 6 in 2024-03-04T16:14:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:14:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:56 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:14:57.089 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: pgmap v546: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-03-04T16:14:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:14:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:14:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4205784795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:14:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: osdmap e277: 8 total, 8 up, 6 in 2024-03-04T16:14:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:14:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:56 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:14:57.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:57.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:57.265 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:57.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:57.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:57.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:57.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:57.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.669 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:57.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:57.770 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.806 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:57.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:57.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:57.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:57.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:57 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:14:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:57 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:14:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:57 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:14:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:57 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:14:58.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:58.079 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:57 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:14:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:57 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:14:58.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.217 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:58.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:58.318 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.354 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:58.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:58.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:58.493 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:58.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:14:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:58.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:14:58.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.824 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:58.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:58.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[29779]: 9.5 scrub ok 2024-03-04T16:14:58.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[29779]: pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:14:58.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:14:58.920 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[38113]: 9.5 scrub ok 2024-03-04T16:14:58.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[38113]: pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:14:58.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:58 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:14:58.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:14:58.931 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:14:58.975 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":545,"stamp":"2024-03-04T16:14:53.539080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8173328758,"num_objects":124905,"num_object_clones":0,"num_object_copies":374715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124905,"num_whiteouts":0,"num_read":2071,"num_read_kb":1917,"num_write":153172,"num_write_kb":8865694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66825,"ondisk_log_size":66825,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":27108064,"kb_used_data":24008220,"kb_used_omap":9,"kb_used_meta":3099638,"kb_avail":535362848,"statfs":{"total":575970213888,"available":548211556352,"internally_reserved":0,"allocated":24584417280,"data_stored":24581918779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":3174029991},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-774045696,"num_objects":-11811,"num_object_clones":0,"num_object_copies":-35433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11811,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11811,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002066"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":459,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566445+0000","last_change":"2024-03-04T16:13:18.470267+0000","last_active":"2024-03-04T16:14:52.566445+0000","last_peered":"2024-03-04T16:14:52.566445+0000","last_clean":"2024-03-04T16:14:52.566445+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:14:52.566445+0000","last_undegraded":"2024-03-04T16:14:52.566445+0000","last_fullsized":"2024-03-04T16:14:52.566445+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:18.470133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.470133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":456,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566350+0000","last_change":"2024-03-04T16:12:19.579690+0000","last_active":"2024-03-04T16:14:52.566350+0000","last_peered":"2024-03-04T16:14:52.566350+0000","last_clean":"2024-03-04T16:14:52.566350+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:14:52.566350+0000","last_undegraded":"2024-03-04T16:14:52.566350+0000","last_fullsized":"2024-03-04T16:14:52.566350+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:19.579557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:12:19.579557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":320,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562838+0000","last_change":"2024-03-04T16:13:53.441131+0000","last_active":"2024-03-04T16:14:52.562838+0000","last_peered":"2024-03-04T16:14:52.562838+0000","last_clean":"2024-03-04T16:14:52.562838+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:14:52.562838+0000","last_undegraded":"2024-03-04T16:14:52.562838+0000","last_fullsized":"2024-03-04T16:14:52.562838+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.441019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.441019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.500140+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":509,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566306+0000","last_change":"2024-03-04T16:14:45.600841+0000","last_active":"2024-03-04T16:14:52.566306+0000","last_peered":"2024-03-04T16:14:52.566306+0000","last_clean":"2024-03-04T16:14:52.566306+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:14:52.566306+0000","last_undegraded":"2024-03-04T16:14:52.566306+0000","last_fullsized":"2024-03-04T16:14:52.566306+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:14:45.600687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.600687+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:53.713933+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":453,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512245+0000","last_change":"2024-03-04T16:14:39.888515+0000","last_active":"2024-03-04T16:14:45.512245+0000","last_peered":"2024-03-04T16:14:45.512245+0000","last_clean":"2024-03-04T16:14:45.512245+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:14:45.512245+0000","last_undegraded":"2024-03-04T16:14:45.512245+0000","last_fullsized":"2024-03-04T16:14:45.512245+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:39.888394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:14:39.888394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:40.622315+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":462,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566406+0000","last_change":"2024-03-04T16:12:14.551847+0000","last_active":"2024-03-04T16:14:52.566406+0000","last_peered":"2024-03-04T16:14:52.566406+0000","last_clean":"2024-03-04T16:14:52.566406+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:14:52.566406+0000","last_undegraded":"2024-03-04T16:14:52.566406+0000","last_fullsized":"2024-03-04T16:14:52.566406+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.551417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.551417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":465,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512724+0000","last_change":"2024-03-04T16:13:10.472225+0000","last_active":"2024-03-04T16:14:45.512724+0000","last_peered":"2024-03-04T16:14:45.512724+0000","last_clean":"2024-03-04T16:14:45.512724+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:14:45.512724+0000","last_undegraded":"2024-03-04T16:14:45.512724+0000","last_fullsized":"2024-03-04T16:14:45.512724+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:10.471872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:13:10.471872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":564,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514248+0000","last_change":"2024-03-04T16:12:22.299425+0000","last_active":"2024-03-04T16:14:45.514248+0000","last_peered":"2024-03-04T16:14:45.514248+0000","last_clean":"2024-03-04T16:14:45.514248+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:14:45.514248+0000","last_undegraded":"2024-03-04T16:14:45.514248+0000","last_fullsized":"2024-03-04T16:14:45.514248+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'15","last_scrub_stamp":"2024-03-04T16:12:22.299279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.299279+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":316,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562016+0000","last_change":"2024-03-04T16:14:35.605052+0000","last_active":"2024-03-04T16:14:52.562016+0000","last_peered":"2024-03-04T16:14:52.562016+0000","last_clean":"2024-03-04T16:14:52.562016+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:14:52.562016+0000","last_undegraded":"2024-03-04T16:14:52.562016+0000","last_fullsized":"2024-03-04T16:14:52.562016+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:35.604922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:14:35.604922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:35.945103+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":462,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562029+0000","last_change":"2024-03-04T16:14:49.800277+0000","last_active":"2024-03-04T16:14:52.562029+0000","last_peered":"2024-03-04T16:14:52.562029+0000","last_clean":"2024-03-04T16:14:52.562029+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:14:52.562029+0000","last_undegraded":"2024-03-04T16:14:52.562029+0000","last_fullsized":"2024-03-04T16:14:52.562029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:49.800151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:14:49.800151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:04.850335+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":484,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561137+0000","last_change":"2024-03-04T16:14:42.826084+0000","last_active":"2024-03-04T16:14:52.561137+0000","last_peered":"2024-03-04T16:14:52.561137+0000","last_clean":"2024-03-04T16:14:52.561137+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:14:52.561137+0000","last_undegraded":"2024-03-04T16:14:52.561137+0000","last_fullsized":"2024-03-04T16:14:52.561137+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:42.825972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:42.825972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:50.640176+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":520,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566889+0000","last_change":"2024-03-04T16:11:58.878158+0000","last_active":"2024-03-04T16:14:52.566889+0000","last_peered":"2024-03-04T16:14:52.566889+0000","last_clean":"2024-03-04T16:14:52.566889+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:14:52.566889+0000","last_undegraded":"2024-03-04T16:14:52.566889+0000","last_fullsized":"2024-03-04T16:14:52.566889+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-03-04T16:11:58.878047+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:11:58.878047+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":466,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562118+0000","last_change":"2024-03-04T16:13:59.118849+0000","last_active":"2024-03-04T16:14:52.562118+0000","last_peered":"2024-03-04T16:14:52.562118+0000","last_clean":"2024-03-04T16:14:52.562118+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:14:52.562118+0000","last_undegraded":"2024-03-04T16:14:52.562118+0000","last_fullsized":"2024-03-04T16:14:52.562118+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:59.118720+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.118720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.495900+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564635+0000","last_change":"2024-03-04T16:14:48.045896+0000","last_active":"2024-03-04T16:14:52.564635+0000","last_peered":"2024-03-04T16:14:52.564635+0000","last_clean":"2024-03-04T16:14:52.564635+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:14:52.564635+0000","last_undegraded":"2024-03-04T16:14:52.564635+0000","last_fullsized":"2024-03-04T16:14:52.564635+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:48.043618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:48.043618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:13.276150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560124+0000","last_change":"2024-03-04T16:13:56.116452+0000","last_active":"2024-03-04T16:14:52.560124+0000","last_peered":"2024-03-04T16:14:52.560124+0000","last_clean":"2024-03-04T16:14:52.560124+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:14:52.560124+0000","last_undegraded":"2024-03-04T16:14:52.560124+0000","last_fullsized":"2024-03-04T16:14:52.560124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.116341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.116341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:14:58.120504+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":498,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566950+0000","last_change":"2024-03-04T16:12:15.539669+0000","last_active":"2024-03-04T16:14:52.566950+0000","last_peered":"2024-03-04T16:14:52.566950+0000","last_clean":"2024-03-04T16:14:52.566950+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:14:52.566950+0000","last_undegraded":"2024-03-04T16:14:52.566950+0000","last_fullsized":"2024-03-04T16:14:52.566950+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'7","last_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.539528+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":458,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567468+0000","last_change":"2024-03-04T16:12:20.525886+0000","last_active":"2024-03-04T16:14:52.567468+0000","last_peered":"2024-03-04T16:14:52.567468+0000","last_clean":"2024-03-04T16:14:52.567468+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:14:52.567468+0000","last_undegraded":"2024-03-04T16:14:52.567468+0000","last_fullsized":"2024-03-04T16:14:52.567468+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:20.525761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.525761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":463,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566470+0000","last_change":"2024-03-04T16:13:03.552450+0000","last_active":"2024-03-04T16:14:52.566470+0000","last_peered":"2024-03-04T16:14:52.566470+0000","last_clean":"2024-03-04T16:14:52.566470+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:14:52.566470+0000","last_undegraded":"2024-03-04T16:14:52.566470+0000","last_fullsized":"2024-03-04T16:14:52.566470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:03.552289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:03.552289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":307,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563348+0000","last_change":"2024-03-04T16:14:36.635399+0000","last_active":"2024-03-04T16:14:52.563348+0000","last_peered":"2024-03-04T16:14:52.563348+0000","last_clean":"2024-03-04T16:14:52.563348+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:14:52.563348+0000","last_undegraded":"2024-03-04T16:14:52.563348+0000","last_fullsized":"2024-03-04T16:14:52.563348+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:36.635123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:36.635123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:37.136584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"250'28","reported_seq":278,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567810+0000","last_change":"2024-03-04T16:14:36.667204+0000","last_active":"2024-03-04T16:14:52.567810+0000","last_peered":"2024-03-04T16:14:52.567810+0000","last_clean":"2024-03-04T16:14:52.567810+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:14:52.567810+0000","last_undegraded":"2024-03-04T16:14:52.567810+0000","last_fullsized":"2024-03-04T16:14:52.567810+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'28","last_scrub_stamp":"2024-03-04T16:14:36.666970+0000","last_deep_scrub":"250'28","last_deep_scrub_stamp":"2024-03-04T16:13:18.621837+0000","last_clean_scrub_stamp":"2024-03-04T16:14:36.666970+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:56.196752+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":446,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.515161+0000","last_change":"2024-03-04T16:10:29.352325+0000","last_active":"2024-03-04T16:14:45.515161+0000","last_peered":"2024-03-04T16:14:45.515161+0000","last_clean":"2024-03-04T16:14:45.515161+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:14:45.515161+0000","last_undegraded":"2024-03-04T16:14:45.515161+0000","last_fullsized":"2024-03-04T16:14:45.515161+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:09:59.066708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:09:59.066708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":469,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567812+0000","last_change":"2024-03-04T16:13:58.335444+0000","last_active":"2024-03-04T16:14:52.567812+0000","last_peered":"2024-03-04T16:14:52.567812+0000","last_clean":"2024-03-04T16:14:52.567812+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:14:52.567812+0000","last_undegraded":"2024-03-04T16:14:52.567812+0000","last_fullsized":"2024-03-04T16:14:52.567812+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.334939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.334939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:10.215128+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":477,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566996+0000","last_change":"2024-03-04T16:13:17.426445+0000","last_active":"2024-03-04T16:14:52.566996+0000","last_peered":"2024-03-04T16:14:52.566996+0000","last_clean":"2024-03-04T16:14:52.566996+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:14:52.566996+0000","last_undegraded":"2024-03-04T16:14:52.566996+0000","last_fullsized":"2024-03-04T16:14:52.566996+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.426276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.426276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":495,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513095+0000","last_change":"2024-03-04T16:14:38.765045+0000","last_active":"2024-03-04T16:14:45.513095+0000","last_peered":"2024-03-04T16:14:45.513095+0000","last_clean":"2024-03-04T16:14:45.513095+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:14:45.513095+0000","last_undegraded":"2024-03-04T16:14:45.513095+0000","last_fullsized":"2024-03-04T16:14:45.513095+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:14:38.764896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:38.764896+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:44.485284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564462+0000","last_change":"2024-03-04T16:13:55.908132+0000","last_active":"2024-03-04T16:14:52.564462+0000","last_peered":"2024-03-04T16:14:52.564462+0000","last_clean":"2024-03-04T16:14:52.564462+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:14:52.564462+0000","last_undegraded":"2024-03-04T16:14:52.564462+0000","last_fullsized":"2024-03-04T16:14:52.564462+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:13:55.907870+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.907870+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:03.802762+0000","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":427,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567522+0000","last_change":"2024-03-04T16:10:29.331975+0000","last_active":"2024-03-04T16:14:52.567522+0000","last_peered":"2024-03-04T16:14:52.567522+0000","last_clean":"2024-03-04T16:14:52.567522+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:14:52.567522+0000","last_undegraded":"2024-03-04T16:14:52.567522+0000","last_fullsized":"2024-03-04T16:14:52.567522+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9","last_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_deep_scrub":"124'9","last_deep_scrub_stamp":"2024-03-04T16:10:07.889828+0000","last_clean_scrub_stamp":"2024-03-04T16:10:07.889828+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":473,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566605+0000","last_change":"2024-03-04T16:14:46.696734+0000","last_active":"2024-03-04T16:14:52.566605+0000","last_peered":"2024-03-04T16:14:52.566605+0000","last_clean":"2024-03-04T16:14:52.566605+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:14:52.566605+0000","last_undegraded":"2024-03-04T16:14:52.566605+0000","last_fullsized":"2024-03-04T16:14:52.566605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:14:46.696591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:46.696591+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:51.379434+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":343,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562118+0000","last_change":"2024-03-04T16:14:51.519605+0000","last_active":"2024-03-04T16:14:52.562118+0000","last_peered":"2024-03-04T16:14:52.562118+0000","last_clean":"2024-03-04T16:14:52.562118+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:14:52.562118+0000","last_undegraded":"2024-03-04T16:14:52.562118+0000","last_fullsized":"2024-03-04T16:14:52.562118+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:14:51.519351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.519351+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:54.422240+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":459,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566621+0000","last_change":"2024-03-04T16:14:45.699552+0000","last_active":"2024-03-04T16:14:52.566621+0000","last_peered":"2024-03-04T16:14:52.566621+0000","last_clean":"2024-03-04T16:14:52.566621+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:14:52.566621+0000","last_undegraded":"2024-03-04T16:14:52.566621+0000","last_fullsized":"2024-03-04T16:14:52.566621+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:45.699386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.699386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:06.740604+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":92,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512536+0000","last_change":"2024-03-04T16:12:23.623101+0000","last_active":"2024-03-04T16:14:45.512536+0000","last_peered":"2024-03-04T16:14:45.512536+0000","last_clean":"2024-03-04T16:14:45.512536+0000","last_became_active":"2024-03-04T16:12:23.622602+0000","last_became_peered":"2024-03-04T16:12:23.622602+0000","last_unstale":"2024-03-04T16:14:45.512536+0000","last_undegraded":"2024-03-04T16:14:45.512536+0000","last_fullsized":"2024-03-04T16:14:45.512536+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:02.487604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:02.487604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":471,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513610+0000","last_change":"2024-03-04T16:13:19.603451+0000","last_active":"2024-03-04T16:14:45.513610+0000","last_peered":"2024-03-04T16:14:45.513610+0000","last_clean":"2024-03-04T16:14:45.513610+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:14:45.513610+0000","last_undegraded":"2024-03-04T16:14:45.513610+0000","last_fullsized":"2024-03-04T16:14:45.513610+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.603191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.603191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":246,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561102+0000","last_change":"2024-03-04T16:12:22.748187+0000","last_active":"2024-03-04T16:14:52.561102+0000","last_peered":"2024-03-04T16:14:52.561102+0000","last_clean":"2024-03-04T16:14:52.561102+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:14:52.561102+0000","last_undegraded":"2024-03-04T16:14:52.561102+0000","last_fullsized":"2024-03-04T16:14:52.561102+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:12:22.748049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:12:22.748049+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":474,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564838+0000","last_change":"2024-03-04T16:14:51.078251+0000","last_active":"2024-03-04T16:14:52.564838+0000","last_peered":"2024-03-04T16:14:52.564838+0000","last_clean":"2024-03-04T16:14:52.564838+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:14:52.564838+0000","last_undegraded":"2024-03-04T16:14:52.564838+0000","last_fullsized":"2024-03-04T16:14:52.564838+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:51.078051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.078051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:10.013473+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":183,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567443+0000","last_change":"2024-03-04T16:13:12.406817+0000","last_active":"2024-03-04T16:14:52.567443+0000","last_peered":"2024-03-04T16:14:52.567443+0000","last_clean":"2024-03-04T16:14:52.567443+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:14:52.567443+0000","last_undegraded":"2024-03-04T16:14:52.567443+0000","last_fullsized":"2024-03-04T16:14:52.567443+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.406690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.406690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":308,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562784+0000","last_change":"2024-03-04T16:14:05.418615+0000","last_active":"2024-03-04T16:14:52.562784+0000","last_peered":"2024-03-04T16:14:52.562784+0000","last_clean":"2024-03-04T16:14:52.562784+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:14:52.562784+0000","last_undegraded":"2024-03-04T16:14:52.562784+0000","last_fullsized":"2024-03-04T16:14:52.562784+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:05.418476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:05.418476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.527281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513695+0000","last_change":"2024-03-04T16:14:40.731541+0000","last_active":"2024-03-04T16:14:45.513695+0000","last_peered":"2024-03-04T16:14:45.513695+0000","last_clean":"2024-03-04T16:14:45.513695+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:14:45.513695+0000","last_undegraded":"2024-03-04T16:14:45.513695+0000","last_fullsized":"2024-03-04T16:14:45.513695+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:14:40.731409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:40.731409+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:43.923643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":458,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513302+0000","last_change":"2024-03-04T16:12:14.287844+0000","last_active":"2024-03-04T16:14:45.513302+0000","last_peered":"2024-03-04T16:14:45.513302+0000","last_clean":"2024-03-04T16:14:45.513302+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:14:45.513302+0000","last_undegraded":"2024-03-04T16:14:45.513302+0000","last_fullsized":"2024-03-04T16:14:45.513302+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:12:14.287598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564048+0000","last_change":"2024-03-04T16:13:06.136903+0000","last_active":"2024-03-04T16:14:52.564048+0000","last_peered":"2024-03-04T16:14:52.564048+0000","last_clean":"2024-03-04T16:14:52.564048+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:14:52.564048+0000","last_undegraded":"2024-03-04T16:14:52.564048+0000","last_fullsized":"2024-03-04T16:14:52.564048+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.136769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.136769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean+scrubbing","last_fresh":"2024-03-04T16:14:52.715596+0000","last_change":"2024-03-04T16:14:52.715596+0000","last_active":"2024-03-04T16:14:52.715596+0000","last_peered":"2024-03-04T16:14:52.715596+0000","last_clean":"2024-03-04T16:14:52.715596+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:14:52.715596+0000","last_undegraded":"2024-03-04T16:14:52.715596+0000","last_fullsized":"2024-03-04T16:14:52.715596+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:15.574491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:15.574491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":497,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514214+0000","last_change":"2024-03-04T16:10:29.375879+0000","last_active":"2024-03-04T16:14:45.514214+0000","last_peered":"2024-03-04T16:14:45.514214+0000","last_clean":"2024-03-04T16:14:45.514214+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:14:45.514214+0000","last_undegraded":"2024-03-04T16:14:45.514214+0000","last_fullsized":"2024-03-04T16:14:45.514214+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'11","last_scrub_stamp":"2024-03-04T16:10:03.976596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:03.976596+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":308,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562066+0000","last_change":"2024-03-04T16:14:41.689430+0000","last_active":"2024-03-04T16:14:52.562066+0000","last_peered":"2024-03-04T16:14:52.562066+0000","last_clean":"2024-03-04T16:14:52.562066+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:14:52.562066+0000","last_undegraded":"2024-03-04T16:14:52.562066+0000","last_fullsized":"2024-03-04T16:14:52.562066+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:41.689307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:14:41.689307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:57.687179+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":462,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512155+0000","last_change":"2024-03-04T16:13:04.431970+0000","last_active":"2024-03-04T16:14:45.512155+0000","last_peered":"2024-03-04T16:14:45.512155+0000","last_clean":"2024-03-04T16:14:45.512155+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:14:45.512155+0000","last_undegraded":"2024-03-04T16:14:45.512155+0000","last_fullsized":"2024-03-04T16:14:45.512155+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.431851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.431851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564742+0000","last_change":"2024-03-04T16:13:05.128608+0000","last_active":"2024-03-04T16:14:52.564742+0000","last_peered":"2024-03-04T16:14:52.564742+0000","last_clean":"2024-03-04T16:14:52.564742+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:14:52.564742+0000","last_undegraded":"2024-03-04T16:14:52.564742+0000","last_fullsized":"2024-03-04T16:14:52.564742+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.128508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.128508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":529,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.570294+0000","last_change":"2024-03-04T16:13:14.622077+0000","last_active":"2024-03-04T16:14:52.570294+0000","last_peered":"2024-03-04T16:14:52.570294+0000","last_clean":"2024-03-04T16:14:52.570294+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:14:52.570294+0000","last_undegraded":"2024-03-04T16:14:52.570294+0000","last_fullsized":"2024-03-04T16:14:52.570294+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:14.621919+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.621919+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":452,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513311+0000","last_change":"2024-03-04T16:13:13.530157+0000","last_active":"2024-03-04T16:14:45.513311+0000","last_peered":"2024-03-04T16:14:45.513311+0000","last_clean":"2024-03-04T16:14:45.513311+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:14:45.513311+0000","last_undegraded":"2024-03-04T16:14:45.513311+0000","last_fullsized":"2024-03-04T16:14:45.513311+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.530037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.570054+0000","last_change":"2024-03-04T16:13:12.612940+0000","last_active":"2024-03-04T16:14:52.570054+0000","last_peered":"2024-03-04T16:14:52.570054+0000","last_clean":"2024-03-04T16:14:52.570054+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:14:52.570054+0000","last_undegraded":"2024-03-04T16:14:52.570054+0000","last_fullsized":"2024-03-04T16:14:52.570054+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:12.612806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:12.612806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":478,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567730+0000","last_change":"2024-03-04T16:13:51.392879+0000","last_active":"2024-03-04T16:14:52.567730+0000","last_peered":"2024-03-04T16:14:52.567730+0000","last_clean":"2024-03-04T16:14:52.567730+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:14:52.567730+0000","last_undegraded":"2024-03-04T16:14:52.567730+0000","last_fullsized":"2024-03-04T16:14:52.567730+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:51.392701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:51.392701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.094055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":517,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567122+0000","last_change":"2024-03-04T16:13:01.422169+0000","last_active":"2024-03-04T16:14:52.567122+0000","last_peered":"2024-03-04T16:14:52.567122+0000","last_clean":"2024-03-04T16:14:52.567122+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:14:52.567122+0000","last_undegraded":"2024-03-04T16:14:52.567122+0000","last_fullsized":"2024-03-04T16:14:52.567122+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:01.422054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:01.422054+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":333,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.565237+0000","last_change":"2024-03-04T16:12:03.904682+0000","last_active":"2024-03-04T16:14:52.565237+0000","last_peered":"2024-03-04T16:14:52.565237+0000","last_clean":"2024-03-04T16:14:52.565237+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:14:52.565237+0000","last_undegraded":"2024-03-04T16:14:52.565237+0000","last_fullsized":"2024-03-04T16:14:52.565237+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:03.903765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:03.903765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":638,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561459+0000","last_change":"2024-03-04T16:14:50.750087+0000","last_active":"2024-03-04T16:14:52.561459+0000","last_peered":"2024-03-04T16:14:52.561459+0000","last_clean":"2024-03-04T16:14:52.561459+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:14:52.561459+0000","last_undegraded":"2024-03-04T16:14:52.561459+0000","last_fullsized":"2024-03-04T16:14:52.561459+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:14:50.749848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:50.749848+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:01.169426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":506,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564549+0000","last_change":"2024-03-04T16:13:54.812148+0000","last_active":"2024-03-04T16:14:52.564549+0000","last_peered":"2024-03-04T16:14:52.564549+0000","last_clean":"2024-03-04T16:14:52.564549+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:14:52.564549+0000","last_undegraded":"2024-03-04T16:14:52.564549+0000","last_fullsized":"2024-03-04T16:14:52.564549+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:54.812003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.812003+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:21.284093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":514,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513280+0000","last_change":"2024-03-04T16:14:03.533418+0000","last_active":"2024-03-04T16:14:45.513280+0000","last_peered":"2024-03-04T16:14:45.513280+0000","last_clean":"2024-03-04T16:14:45.513280+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:14:45.513280+0000","last_undegraded":"2024-03-04T16:14:45.513280+0000","last_fullsized":"2024-03-04T16:14:45.513280+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:14:03.533128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:03.533128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:27.853629+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":457,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567618+0000","last_change":"2024-03-04T16:12:04.823911+0000","last_active":"2024-03-04T16:14:52.567618+0000","last_peered":"2024-03-04T16:14:52.567618+0000","last_clean":"2024-03-04T16:14:52.567618+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:14:52.567618+0000","last_undegraded":"2024-03-04T16:14:52.567618+0000","last_fullsized":"2024-03-04T16:14:52.567618+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:04.823773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:04.823773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":608,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514134+0000","last_change":"2024-03-04T16:12:20.287137+0000","last_active":"2024-03-04T16:14:45.514134+0000","last_peered":"2024-03-04T16:14:45.514134+0000","last_clean":"2024-03-04T16:14:45.514134+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:14:45.514134+0000","last_undegraded":"2024-03-04T16:14:45.514134+0000","last_fullsized":"2024-03-04T16:14:45.514134+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:12:20.286793+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:12:20.286793+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":365,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564511+0000","last_change":"2024-03-04T16:13:14.023544+0000","last_active":"2024-03-04T16:14:52.564511+0000","last_peered":"2024-03-04T16:14:52.564511+0000","last_clean":"2024-03-04T16:14:52.564511+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:14:52.564511+0000","last_undegraded":"2024-03-04T16:14:52.564511+0000","last_fullsized":"2024-03-04T16:14:52.564511+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:13:14.023420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:14.023420+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":492,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561425+0000","last_change":"2024-03-04T16:13:05.639843+0000","last_active":"2024-03-04T16:14:52.561425+0000","last_peered":"2024-03-04T16:14:52.561425+0000","last_clean":"2024-03-04T16:14:52.561425+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:14:52.561425+0000","last_undegraded":"2024-03-04T16:14:52.561425+0000","last_fullsized":"2024-03-04T16:14:52.561425+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:05.639731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.639731+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":191,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567926+0000","last_change":"2024-03-04T16:14:50.760982+0000","last_active":"2024-03-04T16:14:52.567926+0000","last_peered":"2024-03-04T16:14:52.567926+0000","last_clean":"2024-03-04T16:14:52.567926+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:14:52.567926+0000","last_undegraded":"2024-03-04T16:14:52.567926+0000","last_fullsized":"2024-03-04T16:14:52.567926+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:50.760883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:50.760883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:59.077239+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":443,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514871+0000","last_change":"2024-03-04T16:10:01.007163+0000","last_active":"2024-03-04T16:14:45.514871+0000","last_peered":"2024-03-04T16:14:45.514871+0000","last_clean":"2024-03-04T16:14:45.514871+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:14:45.514871+0000","last_undegraded":"2024-03-04T16:14:45.514871+0000","last_fullsized":"2024-03-04T16:14:45.514871+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:01.006331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:10:01.006331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564703+0000","last_change":"2024-03-04T16:12:00.921309+0000","last_active":"2024-03-04T16:14:52.564703+0000","last_peered":"2024-03-04T16:14:52.564703+0000","last_clean":"2024-03-04T16:14:52.564703+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:14:52.564703+0000","last_undegraded":"2024-03-04T16:14:52.564703+0000","last_fullsized":"2024-03-04T16:14:52.564703+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.921174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.921174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":327,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562263+0000","last_change":"2024-03-04T16:13:09.675634+0000","last_active":"2024-03-04T16:14:52.562263+0000","last_peered":"2024-03-04T16:14:52.562263+0000","last_clean":"2024-03-04T16:14:52.562263+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:14:52.562263+0000","last_undegraded":"2024-03-04T16:14:52.562263+0000","last_fullsized":"2024-03-04T16:14:52.562263+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:13:09.675534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:09.675534+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":466,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564735+0000","last_change":"2024-03-04T16:12:18.923785+0000","last_active":"2024-03-04T16:14:52.564735+0000","last_peered":"2024-03-04T16:14:52.564735+0000","last_clean":"2024-03-04T16:14:52.564735+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:14:52.564735+0000","last_undegraded":"2024-03-04T16:14:52.564735+0000","last_fullsized":"2024-03-04T16:14:52.564735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:18.923671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:10:07.797282+0000","last_clean_scrub_stamp":"2024-03-04T16:12:18.923671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":97,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":462,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560619+0000","last_change":"2024-03-04T16:13:06.668331+0000","last_active":"2024-03-04T16:14:52.560619+0000","last_peered":"2024-03-04T16:14:52.560619+0000","last_clean":"2024-03-04T16:14:52.560619+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:14:52.560619+0000","last_undegraded":"2024-03-04T16:14:52.560619+0000","last_fullsized":"2024-03-04T16:14:52.560619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:06.668128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:13:06.668128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":377,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562750+0000","last_change":"2024-03-04T16:14:48.552522+0000","last_active":"2024-03-04T16:14:52.562750+0000","last_peered":"2024-03-04T16:14:52.562750+0000","last_clean":"2024-03-04T16:14:52.562750+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:14:52.562750+0000","last_undegraded":"2024-03-04T16:14:52.562750+0000","last_fullsized":"2024-03-04T16:14:52.562750+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:48.552292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:14:48.552292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:16.029206+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":468,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.515077+0000","last_change":"2024-03-04T16:09:56.937473+0000","last_active":"2024-03-04T16:14:45.515077+0000","last_peered":"2024-03-04T16:14:45.515077+0000","last_clean":"2024-03-04T16:14:45.515077+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:14:45.515077+0000","last_undegraded":"2024-03-04T16:14:45.515077+0000","last_fullsized":"2024-03-04T16:14:45.515077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'9","last_scrub_stamp":"2024-03-04T16:09:56.937239+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:09:56.937239+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"276'9806","reported_seq":9884,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:53.287246+0000","last_change":"2024-03-04T16:12:58.261525+0000","last_active":"2024-03-04T16:14:53.287246+0000","last_peered":"2024-03-04T16:14:53.287246+0000","last_clean":"2024-03-04T16:14:53.287246+0000","last_became_active":"2024-03-04T16:12:58.261143+0000","last_became_peered":"2024-03-04T16:12:58.261143+0000","last_unstale":"2024-03-04T16:14:53.287246+0000","last_undegraded":"2024-03-04T16:14:53.287246+0000","last_fullsized":"2024-03-04T16:14:53.287246+0000","mapping_epoch":242,"log_start":"266'5600","ondisk_log_start":"266'5600","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504233984,"num_objects":7694,"num_object_clones":0,"num_object_copies":23082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9806,"num_write_kb":560000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":465,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561309+0000","last_change":"2024-03-04T16:13:04.643016+0000","last_active":"2024-03-04T16:14:52.561309+0000","last_peered":"2024-03-04T16:14:52.561309+0000","last_clean":"2024-03-04T16:14:52.561309+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:14:52.561309+0000","last_undegraded":"2024-03-04T16:14:52.561309+0000","last_fullsized":"2024-03-04T16:14:52.561309+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:04.642875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:04.642875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":508,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.565796+0000","last_change":"2024-03-04T16:12:26.939744+0000","last_active":"2024-03-04T16:14:52.565796+0000","last_peered":"2024-03-04T16:14:52.565796+0000","last_clean":"2024-03-04T16:14:52.565796+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:14:52.565796+0000","last_undegraded":"2024-03-04T16:14:52.565796+0000","last_fullsized":"2024-03-04T16:14:52.565796+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:12:26.939557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.939557+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":540,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560781+0000","last_change":"2024-03-04T16:14:45.845126+0000","last_active":"2024-03-04T16:14:52.560781+0000","last_peered":"2024-03-04T16:14:52.560781+0000","last_clean":"2024-03-04T16:14:52.560781+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:14:52.560781+0000","last_undegraded":"2024-03-04T16:14:52.560781+0000","last_fullsized":"2024-03-04T16:14:52.560781+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:14:45.844973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.844973+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:09.809704+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":628,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.565086+0000","last_change":"2024-03-04T16:13:02.118333+0000","last_active":"2024-03-04T16:14:52.565086+0000","last_peered":"2024-03-04T16:14:52.565086+0000","last_clean":"2024-03-04T16:14:52.565086+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:14:52.565086+0000","last_undegraded":"2024-03-04T16:14:52.565086+0000","last_fullsized":"2024-03-04T16:14:52.565086+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:13:02.118207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:02.118207+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"276'9764","reported_seq":9842,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:53.209784+0000","last_change":"2024-03-04T16:12:58.258448+0000","last_active":"2024-03-04T16:14:53.209784+0000","last_peered":"2024-03-04T16:14:53.209784+0000","last_clean":"2024-03-04T16:14:53.209784+0000","last_became_active":"2024-03-04T16:12:58.258106+0000","last_became_peered":"2024-03-04T16:12:58.258106+0000","last_unstale":"2024-03-04T16:14:53.209784+0000","last_undegraded":"2024-03-04T16:14:53.209784+0000","last_fullsized":"2024-03-04T16:14:53.209784+0000","mapping_epoch":242,"log_start":"269'6200","ondisk_log_start":"269'6200","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506462208,"num_objects":7728,"num_object_clones":0,"num_object_copies":23184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9764,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":448,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512277+0000","last_change":"2024-03-04T16:13:52.548216+0000","last_active":"2024-03-04T16:14:45.512277+0000","last_peered":"2024-03-04T16:14:45.512277+0000","last_clean":"2024-03-04T16:14:45.512277+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:14:45.512277+0000","last_undegraded":"2024-03-04T16:14:45.512277+0000","last_fullsized":"2024-03-04T16:14:45.512277+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.548036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.548036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.053562+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":480,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.565455+0000","last_change":"2024-03-04T16:14:50.034340+0000","last_active":"2024-03-04T16:14:52.565455+0000","last_peered":"2024-03-04T16:14:52.565455+0000","last_clean":"2024-03-04T16:14:52.565455+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:14:52.565455+0000","last_undegraded":"2024-03-04T16:14:52.565455+0000","last_fullsized":"2024-03-04T16:14:52.565455+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:50.034220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:50.034220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:10.744069+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":500,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.568029+0000","last_change":"2024-03-04T16:13:19.438912+0000","last_active":"2024-03-04T16:14:52.568029+0000","last_peered":"2024-03-04T16:14:52.568029+0000","last_clean":"2024-03-04T16:14:52.568029+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:14:52.568029+0000","last_undegraded":"2024-03-04T16:14:52.568029+0000","last_fullsized":"2024-03-04T16:14:52.568029+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:13:19.438774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.438774+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":370,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561876+0000","last_change":"2024-03-04T16:13:55.457911+0000","last_active":"2024-03-04T16:14:52.561876+0000","last_peered":"2024-03-04T16:14:52.561876+0000","last_clean":"2024-03-04T16:14:52.561876+0000","last_became_active":"2024-03-04T16:10:17.477767+0000","last_became_peered":"2024-03-04T16:10:17.477767+0000","last_unstale":"2024-03-04T16:14:52.561876+0000","last_undegraded":"2024-03-04T16:14:52.561876+0000","last_fullsized":"2024-03-04T16:14:52.561876+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.457585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.457585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:05.916401+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"276'9483","reported_seq":9561,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.686131+0000","last_change":"2024-03-04T16:12:58.259241+0000","last_active":"2024-03-04T16:14:52.686131+0000","last_peered":"2024-03-04T16:14:52.686131+0000","last_clean":"2024-03-04T16:14:52.686131+0000","last_became_active":"2024-03-04T16:12:58.258565+0000","last_became_peered":"2024-03-04T16:12:58.258565+0000","last_unstale":"2024-03-04T16:14:52.686131+0000","last_undegraded":"2024-03-04T16:14:52.686131+0000","last_fullsized":"2024-03-04T16:14:52.686131+0000","mapping_epoch":242,"log_start":"264'5100","ondisk_log_start":"264'5100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":502464512,"num_objects":7667,"num_object_clones":0,"num_object_copies":23001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9483,"num_write_kb":548800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563321+0000","last_change":"2024-03-04T16:14:45.047607+0000","last_active":"2024-03-04T16:14:52.563321+0000","last_peered":"2024-03-04T16:14:52.563321+0000","last_clean":"2024-03-04T16:14:52.563321+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:14:52.563321+0000","last_undegraded":"2024-03-04T16:14:52.563321+0000","last_fullsized":"2024-03-04T16:14:52.563321+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:45.047519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.047519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:08.552557+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563588+0000","last_change":"2024-03-04T16:13:19.665876+0000","last_active":"2024-03-04T16:14:52.563588+0000","last_peered":"2024-03-04T16:14:52.563588+0000","last_clean":"2024-03-04T16:14:52.563588+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:14:52.563588+0000","last_undegraded":"2024-03-04T16:14:52.563588+0000","last_fullsized":"2024-03-04T16:14:52.563588+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:19.665604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:19.665604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":478,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564917+0000","last_change":"2024-03-04T16:12:10.817851+0000","last_active":"2024-03-04T16:14:52.564917+0000","last_peered":"2024-03-04T16:14:52.564917+0000","last_clean":"2024-03-04T16:14:52.564917+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:14:52.564917+0000","last_undegraded":"2024-03-04T16:14:52.564917+0000","last_fullsized":"2024-03-04T16:14:52.564917+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:10.817671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:12:10.817671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":279,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.569933+0000","last_change":"2024-03-04T16:13:13.647026+0000","last_active":"2024-03-04T16:14:52.569933+0000","last_peered":"2024-03-04T16:14:52.569933+0000","last_clean":"2024-03-04T16:14:52.569933+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:14:52.569933+0000","last_undegraded":"2024-03-04T16:14:52.569933+0000","last_fullsized":"2024-03-04T16:14:52.569933+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.646813+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"276'9620","reported_seq":9698,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.699809+0000","last_change":"2024-03-04T16:12:58.259225+0000","last_active":"2024-03-04T16:14:52.699809+0000","last_peered":"2024-03-04T16:14:52.699809+0000","last_clean":"2024-03-04T16:14:52.699809+0000","last_became_active":"2024-03-04T16:12:58.258787+0000","last_became_peered":"2024-03-04T16:12:58.258787+0000","last_unstale":"2024-03-04T16:14:52.699809+0000","last_undegraded":"2024-03-04T16:14:52.699809+0000","last_fullsized":"2024-03-04T16:14:52.699809+0000","mapping_epoch":242,"log_start":"264'5200","ondisk_log_start":"264'5200","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4420,"log_dups_size":0,"ondisk_log_size":4420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508297216,"num_objects":7756,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9620,"num_write_kb":556032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":347,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564554+0000","last_change":"2024-03-04T16:13:53.931017+0000","last_active":"2024-03-04T16:14:52.564554+0000","last_peered":"2024-03-04T16:14:52.564554+0000","last_clean":"2024-03-04T16:14:52.564554+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:14:52.564554+0000","last_undegraded":"2024-03-04T16:14:52.564554+0000","last_fullsized":"2024-03-04T16:14:52.564554+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:53.930927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:47.054678+0000","last_clean_scrub_stamp":"2024-03-04T16:13:53.930927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.204703+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":192,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567145+0000","last_change":"2024-03-04T16:14:47.655351+0000","last_active":"2024-03-04T16:14:52.567145+0000","last_peered":"2024-03-04T16:14:52.567145+0000","last_clean":"2024-03-04T16:14:52.567145+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:14:52.567145+0000","last_undegraded":"2024-03-04T16:14:52.567145+0000","last_fullsized":"2024-03-04T16:14:52.567145+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:47.655216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:14:47.655216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:12.193187+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":477,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560209+0000","last_change":"2024-03-04T16:13:07.694160+0000","last_active":"2024-03-04T16:14:52.560209+0000","last_peered":"2024-03-04T16:14:52.560209+0000","last_clean":"2024-03-04T16:14:52.560209+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:14:52.560209+0000","last_undegraded":"2024-03-04T16:14:52.560209+0000","last_fullsized":"2024-03-04T16:14:52.560209+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:07.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:13:07.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":327,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563442+0000","last_change":"2024-03-04T16:13:16.698881+0000","last_active":"2024-03-04T16:14:52.563442+0000","last_peered":"2024-03-04T16:14:52.563442+0000","last_clean":"2024-03-04T16:14:52.563442+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:14:52.563442+0000","last_undegraded":"2024-03-04T16:14:52.563442+0000","last_fullsized":"2024-03-04T16:14:52.563442+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.698648+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"276'9743","reported_seq":9915,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:53.333434+0000","last_change":"2024-03-04T16:14:46.917387+0000","last_active":"2024-03-04T16:14:53.333434+0000","last_peered":"2024-03-04T16:14:53.333434+0000","last_clean":"2024-03-04T16:14:53.333434+0000","last_became_active":"2024-03-04T16:12:58.259075+0000","last_became_peered":"2024-03-04T16:12:58.259075+0000","last_unstale":"2024-03-04T16:14:53.333434+0000","last_undegraded":"2024-03-04T16:14:53.333434+0000","last_fullsized":"2024-03-04T16:14:53.333434+0000","mapping_epoch":242,"log_start":"266'5600","ondisk_log_start":"266'5600","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'9172","last_scrub_stamp":"2024-03-04T16:14:46.917236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:14:46.917236+0000","objects_scrubbed":8454,"log_size":4143,"log_dups_size":0,"ondisk_log_size":4143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:56.980244+0000","scrub_duration":3303,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507838464,"num_objects":7749,"num_object_clones":0,"num_object_copies":23247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7749,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9743,"num_write_kb":559744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":440,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512925+0000","last_change":"2024-03-04T16:10:06.986591+0000","last_active":"2024-03-04T16:14:45.512925+0000","last_peered":"2024-03-04T16:14:45.512925+0000","last_clean":"2024-03-04T16:14:45.512925+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:14:45.512925+0000","last_undegraded":"2024-03-04T16:14:45.512925+0000","last_fullsized":"2024-03-04T16:14:45.512925+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:06.986463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:10:06.986463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":492,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560980+0000","last_change":"2024-03-04T16:12:58.666189+0000","last_active":"2024-03-04T16:14:52.560980+0000","last_peered":"2024-03-04T16:14:52.560980+0000","last_clean":"2024-03-04T16:14:52.560980+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:14:52.560980+0000","last_undegraded":"2024-03-04T16:14:52.560980+0000","last_fullsized":"2024-03-04T16:14:52.560980+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:12:58.666070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:12:58.666070+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":490,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566668+0000","last_change":"2024-03-04T16:12:25.567411+0000","last_active":"2024-03-04T16:14:52.566668+0000","last_peered":"2024-03-04T16:14:52.566668+0000","last_clean":"2024-03-04T16:14:52.566668+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:14:52.566668+0000","last_undegraded":"2024-03-04T16:14:52.566668+0000","last_fullsized":"2024-03-04T16:14:52.566668+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:25.567295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:12:25.567295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":736,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.568050+0000","last_change":"2024-03-04T16:14:44.638225+0000","last_active":"2024-03-04T16:14:52.568050+0000","last_peered":"2024-03-04T16:14:52.568050+0000","last_clean":"2024-03-04T16:14:52.568050+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:14:52.568050+0000","last_undegraded":"2024-03-04T16:14:52.568050+0000","last_fullsized":"2024-03-04T16:14:52.568050+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:14:44.638072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:44.638072+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:58.156700+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":308,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562623+0000","last_change":"2024-03-04T16:13:52.478892+0000","last_active":"2024-03-04T16:14:52.562623+0000","last_peered":"2024-03-04T16:14:52.562623+0000","last_clean":"2024-03-04T16:14:52.562623+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:14:52.562623+0000","last_undegraded":"2024-03-04T16:14:52.562623+0000","last_fullsized":"2024-03-04T16:14:52.562623+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:52.478776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:52.478776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:15.051799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"276'9684","reported_seq":9762,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.666315+0000","last_change":"2024-03-04T16:12:58.263282+0000","last_active":"2024-03-04T16:14:52.666315+0000","last_peered":"2024-03-04T16:14:52.666315+0000","last_clean":"2024-03-04T16:14:52.666315+0000","last_became_active":"2024-03-04T16:12:58.262950+0000","last_became_peered":"2024-03-04T16:12:58.262950+0000","last_unstale":"2024-03-04T16:14:52.666315+0000","last_undegraded":"2024-03-04T16:14:52.666315+0000","last_fullsized":"2024-03-04T16:14:52.666315+0000","mapping_epoch":242,"log_start":"264'5300","ondisk_log_start":"264'5300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506462208,"num_objects":7728,"num_object_clones":0,"num_object_copies":23184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9684,"num_write_kb":557184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":466,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.565009+0000","last_change":"2024-03-04T16:13:20.064259+0000","last_active":"2024-03-04T16:14:52.565009+0000","last_peered":"2024-03-04T16:14:52.565009+0000","last_clean":"2024-03-04T16:14:52.565009+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:14:52.565009+0000","last_undegraded":"2024-03-04T16:14:52.565009+0000","last_fullsized":"2024-03-04T16:14:52.565009+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:20.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:20.064148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":485,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566802+0000","last_change":"2024-03-04T16:14:39.591492+0000","last_active":"2024-03-04T16:14:52.566802+0000","last_peered":"2024-03-04T16:14:52.566802+0000","last_clean":"2024-03-04T16:14:52.566802+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:14:52.566802+0000","last_undegraded":"2024-03-04T16:14:52.566802+0000","last_fullsized":"2024-03-04T16:14:52.566802+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:39.591423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:39.591423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:45.922751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":189,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513408+0000","last_change":"2024-03-04T16:14:01.534096+0000","last_active":"2024-03-04T16:14:45.513408+0000","last_peered":"2024-03-04T16:14:45.513408+0000","last_clean":"2024-03-04T16:14:45.513408+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:14:45.513408+0000","last_undegraded":"2024-03-04T16:14:45.513408+0000","last_fullsized":"2024-03-04T16:14:45.513408+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.533961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.533961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:08.148103+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":116,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566743+0000","last_change":"2024-03-04T16:14:42.634175+0000","last_active":"2024-03-04T16:14:52.566743+0000","last_peered":"2024-03-04T16:14:52.566743+0000","last_clean":"2024-03-04T16:14:52.566743+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:14:52.566743+0000","last_undegraded":"2024-03-04T16:14:52.566743+0000","last_fullsized":"2024-03-04T16:14:52.566743+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:14:42.633879+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:14:42.633879+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:05.625487+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":192,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560840+0000","last_change":"2024-03-04T16:14:37.963752+0000","last_active":"2024-03-04T16:14:52.560840+0000","last_peered":"2024-03-04T16:14:52.560840+0000","last_clean":"2024-03-04T16:14:52.560840+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:14:52.560840+0000","last_undegraded":"2024-03-04T16:14:52.560840+0000","last_fullsized":"2024-03-04T16:14:52.560840+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:37.963631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:37.963631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:56.662684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"274'9183","reported_seq":9257,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:48.612765+0000","last_change":"2024-03-04T16:12:58.256227+0000","last_active":"2024-03-04T16:14:48.612765+0000","last_peered":"2024-03-04T16:14:48.612765+0000","last_clean":"2024-03-04T16:14:48.612765+0000","last_became_active":"2024-03-04T16:12:58.255897+0000","last_became_peered":"2024-03-04T16:12:58.255897+0000","last_unstale":"2024-03-04T16:14:48.612765+0000","last_undegraded":"2024-03-04T16:14:48.612765+0000","last_fullsized":"2024-03-04T16:14:48.612765+0000","mapping_epoch":242,"log_start":"264'5100","ondisk_log_start":"264'5100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525402112,"num_objects":8017,"num_object_clones":0,"num_object_copies":24051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":550400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":459,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.560884+0000","last_change":"2024-03-04T16:12:23.816872+0000","last_active":"2024-03-04T16:14:52.560884+0000","last_peered":"2024-03-04T16:14:52.560884+0000","last_clean":"2024-03-04T16:14:52.560884+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:14:52.560884+0000","last_undegraded":"2024-03-04T16:14:52.560884+0000","last_fullsized":"2024-03-04T16:14:52.560884+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:23.816724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:23.816724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":779,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563534+0000","last_change":"2024-03-04T16:12:59.102019+0000","last_active":"2024-03-04T16:14:52.563534+0000","last_peered":"2024-03-04T16:14:52.563534+0000","last_clean":"2024-03-04T16:14:52.563534+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:14:52.563534+0000","last_undegraded":"2024-03-04T16:14:52.563534+0000","last_fullsized":"2024-03-04T16:14:52.563534+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:12:59.101913+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:12:59.101913+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":466,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512484+0000","last_change":"2024-03-04T16:13:56.375663+0000","last_active":"2024-03-04T16:14:45.512484+0000","last_peered":"2024-03-04T16:14:45.512484+0000","last_clean":"2024-03-04T16:14:45.512484+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:14:45.512484+0000","last_undegraded":"2024-03-04T16:14:45.512484+0000","last_fullsized":"2024-03-04T16:14:45.512484+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:56.375405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:56.375405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:24.685421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":316,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567325+0000","last_change":"2024-03-04T16:14:46.647014+0000","last_active":"2024-03-04T16:14:52.567325+0000","last_peered":"2024-03-04T16:14:52.567325+0000","last_clean":"2024-03-04T16:14:52.567325+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:14:52.567325+0000","last_undegraded":"2024-03-04T16:14:52.567325+0000","last_fullsized":"2024-03-04T16:14:52.567325+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:14:46.646882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:46.646882+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:49.553764+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564568+0000","last_change":"2024-03-04T16:12:09.805823+0000","last_active":"2024-03-04T16:14:52.564568+0000","last_peered":"2024-03-04T16:14:52.564568+0000","last_clean":"2024-03-04T16:14:52.564568+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:14:52.564568+0000","last_undegraded":"2024-03-04T16:14:52.564568+0000","last_fullsized":"2024-03-04T16:14:52.564568+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:09.805715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:12:09.805715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"276'9542","reported_seq":9713,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:53.341091+0000","last_change":"2024-03-04T16:14:45.750495+0000","last_active":"2024-03-04T16:14:53.341091+0000","last_peered":"2024-03-04T16:14:53.341091+0000","last_clean":"2024-03-04T16:14:53.341091+0000","last_became_active":"2024-03-04T16:12:58.258386+0000","last_became_peered":"2024-03-04T16:12:58.258386+0000","last_unstale":"2024-03-04T16:14:53.341091+0000","last_undegraded":"2024-03-04T16:14:53.341091+0000","last_fullsized":"2024-03-04T16:14:53.341091+0000","mapping_epoch":242,"log_start":"266'5400","ondisk_log_start":"266'5400","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'8886","last_scrub_stamp":"2024-03-04T16:14:45.750270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.750270+0000","objects_scrubbed":8394,"log_size":4142,"log_dups_size":0,"ondisk_log_size":4142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:13.516432+0000","scrub_duration":3137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500170752,"num_objects":7632,"num_object_clones":0,"num_object_copies":22896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9542,"num_write_kb":549568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":468,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567663+0000","last_change":"2024-03-04T16:14:51.649897+0000","last_active":"2024-03-04T16:14:52.567663+0000","last_peered":"2024-03-04T16:14:52.567663+0000","last_clean":"2024-03-04T16:14:52.567663+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:14:52.567663+0000","last_undegraded":"2024-03-04T16:14:52.567663+0000","last_fullsized":"2024-03-04T16:14:52.567663+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:51.649675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.649675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:58.514622+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":316,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562109+0000","last_change":"2024-03-04T16:13:50.496078+0000","last_active":"2024-03-04T16:14:52.562109+0000","last_peered":"2024-03-04T16:14:52.562109+0000","last_clean":"2024-03-04T16:14:52.562109+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:14:52.562109+0000","last_undegraded":"2024-03-04T16:14:52.562109+0000","last_fullsized":"2024-03-04T16:14:52.562109+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:50.495814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:50.495814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.556381+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1216,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563436+0000","last_change":"2024-03-04T16:13:57.825817+0000","last_active":"2024-03-04T16:14:52.563436+0000","last_peered":"2024-03-04T16:14:52.563436+0000","last_clean":"2024-03-04T16:14:52.563436+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:14:52.563436+0000","last_undegraded":"2024-03-04T16:14:52.563436+0000","last_fullsized":"2024-03-04T16:14:52.563436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:13:57.825680+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:13:57.825680+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.150143+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":722,"num_read_kb":717,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":360,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561910+0000","last_change":"2024-03-04T16:13:15.734217+0000","last_active":"2024-03-04T16:14:52.561910+0000","last_peered":"2024-03-04T16:14:52.561910+0000","last_clean":"2024-03-04T16:14:52.561910+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:14:52.561910+0000","last_undegraded":"2024-03-04T16:14:52.561910+0000","last_fullsized":"2024-03-04T16:14:52.561910+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:13:15.734074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.734074+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561311+0000","last_change":"2024-03-04T16:14:06.002348+0000","last_active":"2024-03-04T16:14:52.561311+0000","last_peered":"2024-03-04T16:14:52.561311+0000","last_clean":"2024-03-04T16:14:52.561311+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:14:52.561311+0000","last_undegraded":"2024-03-04T16:14:52.561311+0000","last_fullsized":"2024-03-04T16:14:52.561311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:06.002229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.002229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:30.238482+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"275'9547","reported_seq":9625,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.563641+0000","last_change":"2024-03-04T16:12:58.261720+0000","last_active":"2024-03-04T16:14:52.563641+0000","last_peered":"2024-03-04T16:14:52.563641+0000","last_clean":"2024-03-04T16:14:52.563641+0000","last_became_active":"2024-03-04T16:12:58.261282+0000","last_became_peered":"2024-03-04T16:12:58.261282+0000","last_unstale":"2024-03-04T16:14:52.563641+0000","last_undegraded":"2024-03-04T16:14:52.563641+0000","last_fullsized":"2024-03-04T16:14:52.563641+0000","mapping_epoch":242,"log_start":"266'5400","ondisk_log_start":"266'5400","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4147,"log_dups_size":0,"ondisk_log_size":4147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506527744,"num_objects":7729,"num_object_clones":0,"num_object_copies":23187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9547,"num_write_kb":552832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566402+0000","last_change":"2024-03-04T16:13:59.616838+0000","last_active":"2024-03-04T16:14:52.566402+0000","last_peered":"2024-03-04T16:14:52.566402+0000","last_clean":"2024-03-04T16:14:52.566402+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:14:52.566402+0000","last_undegraded":"2024-03-04T16:14:52.566402+0000","last_fullsized":"2024-03-04T16:14:52.566402+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:59.616709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:59.616709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:13.656637+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567907+0000","last_change":"2024-03-04T16:13:05.550936+0000","last_active":"2024-03-04T16:14:52.567907+0000","last_peered":"2024-03-04T16:14:52.567907+0000","last_clean":"2024-03-04T16:14:52.567907+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:14:52.567907+0000","last_undegraded":"2024-03-04T16:14:52.567907+0000","last_fullsized":"2024-03-04T16:14:52.567907+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:05.550829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.550829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":446,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562054+0000","last_change":"2024-03-04T16:12:17.635817+0000","last_active":"2024-03-04T16:14:52.562054+0000","last_peered":"2024-03-04T16:14:52.562054+0000","last_clean":"2024-03-04T16:14:52.562054+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:14:52.562054+0000","last_undegraded":"2024-03-04T16:14:52.562054+0000","last_fullsized":"2024-03-04T16:14:52.562054+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'27","last_scrub_stamp":"2024-03-04T16:12:17.635712+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:12:17.635712+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":184,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561078+0000","last_change":"2024-03-04T16:14:38.937930+0000","last_active":"2024-03-04T16:14:52.561078+0000","last_peered":"2024-03-04T16:14:52.561078+0000","last_clean":"2024-03-04T16:14:52.561078+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:14:52.561078+0000","last_undegraded":"2024-03-04T16:14:52.561078+0000","last_fullsized":"2024-03-04T16:14:52.561078+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:38.937184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:38.937184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:57.142413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"276'9719","reported_seq":9797,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.692452+0000","last_change":"2024-03-04T16:12:58.262941+0000","last_active":"2024-03-04T16:14:52.692452+0000","last_peered":"2024-03-04T16:14:52.692452+0000","last_clean":"2024-03-04T16:14:52.692452+0000","last_became_active":"2024-03-04T16:12:58.262611+0000","last_became_peered":"2024-03-04T16:12:58.262611+0000","last_unstale":"2024-03-04T16:14:52.692452+0000","last_undegraded":"2024-03-04T16:14:52.692452+0000","last_fullsized":"2024-03-04T16:14:52.692452+0000","mapping_epoch":242,"log_start":"264'5300","ondisk_log_start":"264'5300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514260992,"num_objects":7847,"num_object_clones":0,"num_object_copies":23541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9719,"num_write_kb":562112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":475,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562212+0000","last_change":"2024-03-04T16:14:01.478625+0000","last_active":"2024-03-04T16:14:52.562212+0000","last_peered":"2024-03-04T16:14:52.562212+0000","last_clean":"2024-03-04T16:14:52.562212+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:14:52.562212+0000","last_undegraded":"2024-03-04T16:14:52.562212+0000","last_fullsized":"2024-03-04T16:14:52.562212+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:01.477978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:01.477978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:23.643549+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":483,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561229+0000","last_change":"2024-03-04T16:13:16.814942+0000","last_active":"2024-03-04T16:14:52.561229+0000","last_peered":"2024-03-04T16:14:52.561229+0000","last_clean":"2024-03-04T16:14:52.561229+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:14:52.561229+0000","last_undegraded":"2024-03-04T16:14:52.561229+0000","last_fullsized":"2024-03-04T16:14:52.561229+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.814825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.814825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":503,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.570147+0000","last_change":"2024-03-04T16:13:16.606983+0000","last_active":"2024-03-04T16:14:52.570147+0000","last_peered":"2024-03-04T16:14:52.570147+0000","last_clean":"2024-03-04T16:14:52.570147+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:14:52.570147+0000","last_undegraded":"2024-03-04T16:14:52.570147+0000","last_fullsized":"2024-03-04T16:14:52.570147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.605307+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":266,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562175+0000","last_change":"2024-03-04T16:13:58.080312+0000","last_active":"2024-03-04T16:14:52.562175+0000","last_peered":"2024-03-04T16:14:52.562175+0000","last_clean":"2024-03-04T16:14:52.562175+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:14:52.562175+0000","last_undegraded":"2024-03-04T16:14:52.562175+0000","last_fullsized":"2024-03-04T16:14:52.562175+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:13:58.080163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.080163+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:19.165616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"274'9356","reported_seq":9430,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:48.561714+0000","last_change":"2024-03-04T16:12:58.259682+0000","last_active":"2024-03-04T16:14:48.561714+0000","last_peered":"2024-03-04T16:14:48.561714+0000","last_clean":"2024-03-04T16:14:48.561714+0000","last_became_active":"2024-03-04T16:12:58.259080+0000","last_became_peered":"2024-03-04T16:12:58.259080+0000","last_unstale":"2024-03-04T16:14:48.561714+0000","last_undegraded":"2024-03-04T16:14:48.561714+0000","last_fullsized":"2024-03-04T16:14:48.561714+0000","mapping_epoch":242,"log_start":"264'5300","ondisk_log_start":"264'5300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":542638080,"num_objects":8280,"num_object_clones":0,"num_object_copies":24840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9356,"num_write_kb":564352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":472,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564987+0000","last_change":"2024-03-04T16:11:59.019162+0000","last_active":"2024-03-04T16:14:52.564987+0000","last_peered":"2024-03-04T16:14:52.564987+0000","last_clean":"2024-03-04T16:14:52.564987+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:14:52.564987+0000","last_undegraded":"2024-03-04T16:14:52.564987+0000","last_fullsized":"2024-03-04T16:14:52.564987+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:11:59.019082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.019082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":320,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.562023+0000","last_change":"2024-03-04T16:14:37.613350+0000","last_active":"2024-03-04T16:14:52.562023+0000","last_peered":"2024-03-04T16:14:52.562023+0000","last_clean":"2024-03-04T16:14:52.562023+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:14:52.562023+0000","last_undegraded":"2024-03-04T16:14:52.562023+0000","last_fullsized":"2024-03-04T16:14:52.562023+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:14:37.613237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:14:37.613237+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:51.805994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":531,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567942+0000","last_change":"2024-03-04T16:12:24.566421+0000","last_active":"2024-03-04T16:14:52.567942+0000","last_peered":"2024-03-04T16:14:52.567942+0000","last_clean":"2024-03-04T16:14:52.567942+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:14:52.567942+0000","last_undegraded":"2024-03-04T16:14:52.567942+0000","last_fullsized":"2024-03-04T16:14:52.567942+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'21","last_scrub_stamp":"2024-03-04T16:12:24.566235+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:12:24.566235+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":605,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514042+0000","last_change":"2024-03-04T16:14:44.790711+0000","last_active":"2024-03-04T16:14:45.514042+0000","last_peered":"2024-03-04T16:14:45.514042+0000","last_clean":"2024-03-04T16:14:45.514042+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:14:45.514042+0000","last_undegraded":"2024-03-04T16:14:45.514042+0000","last_fullsized":"2024-03-04T16:14:45.514042+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:14:44.790580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:44.790580+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:49.509945+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"275'9537","reported_seq":9616,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564626+0000","last_change":"2024-03-04T16:12:58.261655+0000","last_active":"2024-03-04T16:14:52.564626+0000","last_peered":"2024-03-04T16:14:52.564626+0000","last_clean":"2024-03-04T16:14:52.564626+0000","last_became_active":"2024-03-04T16:12:58.261321+0000","last_became_peered":"2024-03-04T16:12:58.261321+0000","last_unstale":"2024-03-04T16:14:52.564626+0000","last_undegraded":"2024-03-04T16:14:52.564626+0000","last_fullsized":"2024-03-04T16:14:52.564626+0000","mapping_epoch":242,"log_start":"266'5400","ondisk_log_start":"266'5400","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":494927896,"num_objects":7553,"num_object_clones":0,"num_object_copies":22659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9537,"num_write_kb":546817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566539+0000","last_change":"2024-03-04T16:13:15.438272+0000","last_active":"2024-03-04T16:14:52.566539+0000","last_peered":"2024-03-04T16:14:52.566539+0000","last_clean":"2024-03-04T16:14:52.566539+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:14:52.566539+0000","last_undegraded":"2024-03-04T16:14:52.566539+0000","last_fullsized":"2024-03-04T16:14:52.566539+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:15.438148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:15.438148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.514472+0000","last_change":"2024-03-04T16:10:10.995362+0000","last_active":"2024-03-04T16:14:45.514472+0000","last_peered":"2024-03-04T16:14:45.514472+0000","last_clean":"2024-03-04T16:14:45.514472+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:14:45.514472+0000","last_undegraded":"2024-03-04T16:14:45.514472+0000","last_fullsized":"2024-03-04T16:14:45.514472+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:10.995124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:10:10.995124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":555,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513984+0000","last_change":"2024-03-04T16:10:14.016867+0000","last_active":"2024-03-04T16:14:45.513984+0000","last_peered":"2024-03-04T16:14:45.513984+0000","last_clean":"2024-03-04T16:14:45.513984+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:14:45.513984+0000","last_undegraded":"2024-03-04T16:14:45.513984+0000","last_fullsized":"2024-03-04T16:14:45.513984+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'18","last_scrub_stamp":"2024-03-04T16:10:14.016578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:10:14.016578+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567731+0000","last_change":"2024-03-04T16:14:42.615690+0000","last_active":"2024-03-04T16:14:52.567731+0000","last_peered":"2024-03-04T16:14:52.567731+0000","last_clean":"2024-03-04T16:14:52.567731+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:14:52.567731+0000","last_undegraded":"2024-03-04T16:14:52.567731+0000","last_fullsized":"2024-03-04T16:14:52.567731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:42.615581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:42.615581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:02.287631+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"274'9147","reported_seq":9221,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:48.605974+0000","last_change":"2024-03-04T16:12:58.259473+0000","last_active":"2024-03-04T16:14:48.605974+0000","last_peered":"2024-03-04T16:14:48.605974+0000","last_clean":"2024-03-04T16:14:48.605974+0000","last_became_active":"2024-03-04T16:12:58.259164+0000","last_became_peered":"2024-03-04T16:12:58.259164+0000","last_unstale":"2024-03-04T16:14:48.605974+0000","last_undegraded":"2024-03-04T16:14:48.605974+0000","last_fullsized":"2024-03-04T16:14:48.605974+0000","mapping_epoch":242,"log_start":"264'5100","ondisk_log_start":"264'5100","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4047,"log_dups_size":0,"ondisk_log_size":4047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524615680,"num_objects":8005,"num_object_clones":0,"num_object_copies":24015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9147,"num_write_kb":548864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":471,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561118+0000","last_change":"2024-03-04T16:14:41.961081+0000","last_active":"2024-03-04T16:14:52.561118+0000","last_peered":"2024-03-04T16:14:52.561118+0000","last_clean":"2024-03-04T16:14:52.561118+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:14:52.561118+0000","last_undegraded":"2024-03-04T16:14:52.561118+0000","last_fullsized":"2024-03-04T16:14:52.561118+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:41.960969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:41.960969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:09.382975+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":610,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567591+0000","last_change":"2024-03-04T16:11:59.819002+0000","last_active":"2024-03-04T16:14:52.567591+0000","last_peered":"2024-03-04T16:14:52.567591+0000","last_clean":"2024-03-04T16:14:52.567591+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:14:52.567591+0000","last_undegraded":"2024-03-04T16:14:52.567591+0000","last_fullsized":"2024-03-04T16:14:52.567591+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:11:59.818881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:11:59.818881+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":460,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513843+0000","last_change":"2024-03-04T16:13:58.414151+0000","last_active":"2024-03-04T16:14:45.513843+0000","last_peered":"2024-03-04T16:14:45.513843+0000","last_clean":"2024-03-04T16:14:45.513843+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:14:45.513843+0000","last_undegraded":"2024-03-04T16:14:45.513843+0000","last_fullsized":"2024-03-04T16:14:45.513843+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:58.413756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:13:58.413756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:09.771485+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"240'27","reported_seq":511,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567136+0000","last_change":"2024-03-04T16:14:47.688864+0000","last_active":"2024-03-04T16:14:52.567136+0000","last_peered":"2024-03-04T16:14:52.567136+0000","last_clean":"2024-03-04T16:14:52.567136+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:14:52.567136+0000","last_undegraded":"2024-03-04T16:14:52.567136+0000","last_fullsized":"2024-03-04T16:14:52.567136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:14:47.688667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:47.688667+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:16:14.012139+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"275'9391","reported_seq":9469,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564489+0000","last_change":"2024-03-04T16:12:58.262002+0000","last_active":"2024-03-04T16:14:52.564489+0000","last_peered":"2024-03-04T16:14:52.564489+0000","last_clean":"2024-03-04T16:14:52.564489+0000","last_became_active":"2024-03-04T16:12:58.261597+0000","last_became_peered":"2024-03-04T16:12:58.261597+0000","last_unstale":"2024-03-04T16:14:52.564489+0000","last_undegraded":"2024-03-04T16:14:52.564489+0000","last_fullsized":"2024-03-04T16:14:52.564489+0000","mapping_epoch":242,"log_start":"266'5300","ondisk_log_start":"266'5300","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498532352,"num_objects":7607,"num_object_clones":0,"num_object_copies":22821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9391,"num_write_kb":543936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":451,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513241+0000","last_change":"2024-03-04T16:12:26.381739+0000","last_active":"2024-03-04T16:14:45.513241+0000","last_peered":"2024-03-04T16:14:45.513241+0000","last_clean":"2024-03-04T16:14:45.513241+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:14:45.513241+0000","last_undegraded":"2024-03-04T16:14:45.513241+0000","last_fullsized":"2024-03-04T16:14:45.513241+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:12:26.381198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:26.381198+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":469,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.561197+0000","last_change":"2024-03-04T16:13:17.814252+0000","last_active":"2024-03-04T16:14:52.561197+0000","last_peered":"2024-03-04T16:14:52.561197+0000","last_clean":"2024-03-04T16:14:52.561197+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:14:52.561197+0000","last_undegraded":"2024-03-04T16:14:52.561197+0000","last_fullsized":"2024-03-04T16:14:52.561197+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:17.814150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:17.814150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":470,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567826+0000","last_change":"2024-03-04T16:13:13.359950+0000","last_active":"2024-03-04T16:14:52.567826+0000","last_peered":"2024-03-04T16:14:52.567826+0000","last_clean":"2024-03-04T16:14:52.567826+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:14:52.567826+0000","last_undegraded":"2024-03-04T16:14:52.567826+0000","last_fullsized":"2024-03-04T16:14:52.567826+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:13.359809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:13:13.359809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":522,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567300+0000","last_change":"2024-03-04T16:14:06.377160+0000","last_active":"2024-03-04T16:14:52.567300+0000","last_peered":"2024-03-04T16:14:52.567300+0000","last_clean":"2024-03-04T16:14:52.567300+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:14:52.567300+0000","last_undegraded":"2024-03-04T16:14:52.567300+0000","last_fullsized":"2024-03-04T16:14:52.567300+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:14:06.377028+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:14:06.377028+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:16.277935+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"276'9515","reported_seq":9593,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:53.219267+0000","last_change":"2024-03-04T16:12:58.261338+0000","last_active":"2024-03-04T16:14:53.219267+0000","last_peered":"2024-03-04T16:14:53.219267+0000","last_clean":"2024-03-04T16:14:53.219267+0000","last_became_active":"2024-03-04T16:12:58.260993+0000","last_became_peered":"2024-03-04T16:12:58.260993+0000","last_unstale":"2024-03-04T16:14:53.219267+0000","last_undegraded":"2024-03-04T16:14:53.219267+0000","last_fullsized":"2024-03-04T16:14:53.219267+0000","mapping_epoch":242,"log_start":"268'5900","ondisk_log_start":"268'5900","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499056640,"num_objects":7615,"num_object_clones":0,"num_object_copies":22845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9515,"num_write_kb":548160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":194,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.513982+0000","last_change":"2024-03-04T16:13:55.518681+0000","last_active":"2024-03-04T16:14:45.513982+0000","last_peered":"2024-03-04T16:14:45.513982+0000","last_clean":"2024-03-04T16:14:45.513982+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:14:45.513982+0000","last_undegraded":"2024-03-04T16:14:45.513982+0000","last_fullsized":"2024-03-04T16:14:45.513982+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:55.518567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:13:55.518567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:14.056169+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":464,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.568137+0000","last_change":"2024-03-04T16:13:11.618623+0000","last_active":"2024-03-04T16:14:52.568137+0000","last_peered":"2024-03-04T16:14:52.568137+0000","last_clean":"2024-03-04T16:14:52.568137+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:14:52.568137+0000","last_undegraded":"2024-03-04T16:14:52.568137+0000","last_fullsized":"2024-03-04T16:14:52.568137+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:11.618510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:13:11.618510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":184,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.566687+0000","last_change":"2024-03-04T16:13:16.430026+0000","last_active":"2024-03-04T16:14:52.566687+0000","last_peered":"2024-03-04T16:14:52.566687+0000","last_clean":"2024-03-04T16:14:52.566687+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:14:52.566687+0000","last_undegraded":"2024-03-04T16:14:52.566687+0000","last_fullsized":"2024-03-04T16:14:52.566687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:13:16.429889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:13:16.429889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":565,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567629+0000","last_change":"2024-03-04T16:13:54.369721+0000","last_active":"2024-03-04T16:14:52.567629+0000","last_peered":"2024-03-04T16:14:52.567629+0000","last_clean":"2024-03-04T16:14:52.567629+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:14:52.567629+0000","last_undegraded":"2024-03-04T16:14:52.567629+0000","last_fullsized":"2024-03-04T16:14:52.567629+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:13:54.369577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:54.369577+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:15:18.639737+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"274'9234","reported_seq":9308,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:48.576259+0000","last_change":"2024-03-04T16:12:58.259516+0000","last_active":"2024-03-04T16:14:48.576259+0000","last_peered":"2024-03-04T16:14:48.576259+0000","last_clean":"2024-03-04T16:14:48.576259+0000","last_became_active":"2024-03-04T16:12:58.259119+0000","last_became_peered":"2024-03-04T16:12:58.259119+0000","last_unstale":"2024-03-04T16:14:48.576259+0000","last_undegraded":"2024-03-04T16:14:48.576259+0000","last_fullsized":"2024-03-04T16:14:48.576259+0000","mapping_epoch":242,"log_start":"264'5200","ondisk_log_start":"264'5200","created":242,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:57.228452+0000","last_clean_scrub_stamp":"2024-03-04T16:12:57.228452+0000","objects_scrubbed":0,"log_size":4034,"log_dups_size":0,"ondisk_log_size":4034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530841600,"num_objects":8100,"num_object_clones":0,"num_object_copies":24300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9234,"num_write_kb":554688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":458,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567571+0000","last_change":"2024-03-04T16:12:00.783064+0000","last_active":"2024-03-04T16:14:52.567571+0000","last_peered":"2024-03-04T16:14:52.567571+0000","last_clean":"2024-03-04T16:14:52.567571+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:14:52.567571+0000","last_undegraded":"2024-03-04T16:14:52.567571+0000","last_fullsized":"2024-03-04T16:14:52.567571+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:12:00.782828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:12:00.782828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":339,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.564855+0000","last_change":"2024-03-04T16:13:18.019746+0000","last_active":"2024-03-04T16:14:52.564855+0000","last_peered":"2024-03-04T16:14:52.564855+0000","last_clean":"2024-03-04T16:14:52.564855+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:14:52.564855+0000","last_undegraded":"2024-03-04T16:14:52.564855+0000","last_fullsized":"2024-03-04T16:14:52.564855+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:13:18.019623+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:13:18.019623+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-04T16:14:45.512148+0000","last_change":"2024-03-04T16:10:02.998838+0000","last_active":"2024-03-04T16:14:45.512148+0000","last_peered":"2024-03-04T16:14:45.512148+0000","last_clean":"2024-03-04T16:14:45.512148+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:14:45.512148+0000","last_undegraded":"2024-03-04T16:14:45.512148+0000","last_fullsized":"2024-03-04T16:14:45.512148+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:10:02.998521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:16.373275+0000","last_clean_scrub_stamp":"2024-03-04T16:10:02.998521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":508,"reported_epoch":276,"state":"active+clean","last_fresh":"2024-03-04T16:14:52.567791+0000","last_change":"2024-03-04T16:13:05.484192+0000","last_active":"2024-03-04T16:14:52.567791+0000","last_peered":"2024-03-04T16:14:52.567791+0000","last_clean":"2024-03-04T16:14:52.567791+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:14:52.567791+0000","last_undegraded":"2024-03-04T16:14:52.567791+0000","last_fullsized":"2024-03-04T16:14:52.567791+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:13:05.484009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:13:05.484009+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8172732440,"num_objects":124707,"num_object_clones":0,"num_object_copies":374121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124707,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152271,"num_write_kb":8863233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24413417472,"data_stored":24413405256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65871,"ondisk_log_size":65871,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":722,"num_read_kb":717,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":731,"ondisk_log_size":731,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070614,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5449000,"kb_used_data":4970780,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":88296152,"statfs":{"total":95995035648,"available":90415259648,"internally_reserved":0,"allocated":5090078720,"data_stored":5089660167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":489683402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364954,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575028,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5506888,"kb_used_data":4930104,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":88238264,"statfs":{"total":95995035648,"available":90355982336,"internally_reserved":0,"allocated":5048426496,"data_stored":5047985237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738552,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3392364,"kb_used_data":2973792,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":90352788,"statfs":{"total":95995035648,"available":92521254912,"internally_reserved":0,"allocated":3045163008,"data_stored":3044740319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411502,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149864,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4875608,"kb_used_data":4186320,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":88869544,"statfs":{"total":95995035648,"available":91002413056,"internally_reserved":0,"allocated":4286791680,"data_stored":4286377643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":705755592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953636,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2387476,"kb_used_data":2013384,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91357676,"statfs":{"total":95995035648,"available":93550260224,"internally_reserved":0,"allocated":2061705216,"data_stored":2061304992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626592,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5496728,"kb_used_data":4933840,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":88248424,"statfs":{"total":95995035648,"available":90366386176,"internally_reserved":0,"allocated":5052252160,"data_stored":5051850421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3013672960,"data_stored":3013672960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5010427904,"data_stored":5010423832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4244832256,"data_stored":4244832256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2038956032,"data_stored":2038956032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":5051387904,"data_stored":5051383832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5054140416,"data_stored":5054136344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:14:58.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-04T16:14:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:58.984 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:58.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:14:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:58 smithi176 ceph-mon[33619]: 9.5 scrub ok 2024-03-04T16:14:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:58 smithi176 ceph-mon[33619]: pgmap v548: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.5 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 730 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:14:59.010 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:58 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:14:59.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:14:59.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:14:59.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:14:59.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:14:59.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:59.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:59.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:59.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:14:59.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:14:59.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:14:59.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:14:59.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[29779]: 6.1f deep-scrub starts 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[29779]: 6.1f deep-scrub ok 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[38113]: 6.1f deep-scrub starts 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:14:59.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:14:59 smithi038 ceph-mon[38113]: 6.1f deep-scrub ok 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192298 7.6 7:6b51513d:::benchmark_data_smithi038_77863_object74030:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.47815713099999, 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.062137957, 2024-03-04T16:14:59.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192298 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288416+0000", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288417+0000", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2300000000000001e-06 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288422+0000", 2024-03-04T16:14:59.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.135e-06 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288433+0000", 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0285000000000001e-05 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.288441+0000", 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999993e-06 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.289307+0000", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086642499999999999 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.289315+0000", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5310000000000004e-06 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.340852+0000", 2024-03-04T16:14:59.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0515369299999999 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.340984+0000", 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013177999999999999 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350502+0000", 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0095186539999999997 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350554+0000", 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1827999999999999e-05 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192320 7.6 7:6d231687:::benchmark_data_smithi038_77863_object74052:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.45516986199999, 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.035261258, 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.776 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192320 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311403+0000", 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311405+0000", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.04e-06 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311411+0000", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0859999999999998e-06 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311423+0000", 2024-03-04T16:14:59.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1046000000000001e-05 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311431+0000", 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7099999999999996e-06 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311487+0000", 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6057999999999998e-05 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.779 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.311496+0000", 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4209999999999995e-06 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341512+0000", 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0300166609999999 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341635+0000", 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012253399999999999 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.341758+0000", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012314800000000001 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346404+0000", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046457019999999998 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346411+0000", 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6899999999999992e-06 2024-03-04T16:14:59.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346508+0000", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6351e-05 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346584+0000", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6139999999999999e-05 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346665+0000", 2024-03-04T16:14:59.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0671000000000001e-05 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192361 7.6 7:620642f3:::benchmark_data_smithi038_77863_object74093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.40731872700002, 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99774576100000001, 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.783 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192361 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359254+0000", 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359255+0000", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.23e-07 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359258+0000", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2179999999999999e-06 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359268+0000", 2024-03-04T16:14:59.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1220000000000001e-06 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359273+0000", 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.519e-06 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359428+0000", 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000154499 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.359436+0000", 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8859999999999993e-06 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346702+0000", 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98726572000000001 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346789+0000", 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6763999999999998e-05 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.346861+0000", 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2237999999999995e-05 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350594+0000", 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037329300000000002 2024-03-04T16:14:59.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350651+0000", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6870999999999998e-05 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356749+0000", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006097585 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356860+0000", 2024-03-04T16:14:59.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111586 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357000+0000", 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014009999999999999 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192379 7.6 7:642a4bb6:::benchmark_data_smithi038_77863_object74111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:14:59.790 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.37404547, 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96479910099999999, 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192379 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392528+0000", 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392530+0000", 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4990000000000001e-06 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392538+0000", 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0099999999999995e-06 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392554+0000", 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5384e-05 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392562+0000", 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3969999999999994e-06 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392629+0000", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6717999999999994e-05 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.392641+0000", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2265e-05 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350689+0000", 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95804780499999997 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350784+0000", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4969000000000005e-05 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.350892+0000", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108572 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.356719+0000", 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0058263000000000004 2024-03-04T16:14:59.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357015+0000", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000296545 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357061+0000", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5828000000000002e-05 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357223+0000", 2024-03-04T16:14:59.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016149400000000001 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357327+0000", 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010431500000000001 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192415 7.6 7:6f2667ec:::benchmark_data_smithi038_77863_object74147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.33863476099998, 2024-03-04T16:14:59.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93521214699999999, 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.798 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192415 2024-03-04T16:14:59.798 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427938+0000", 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427940+0000", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.083e-06 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427944+0000", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5190000000000003e-06 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427961+0000", 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7326e-05 2024-03-04T16:14:59.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.427969+0000", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4019999999999996e-06 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.428022+0000", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3430000000000002e-05 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.428030+0000", 2024-03-04T16:14:59.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1249999999999993e-06 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353490+0000", 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92545976799999996 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353635+0000", 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000145358 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.353861+0000", 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000225882 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357393+0000", 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035317090000000001 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357434+0000", 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1405000000000001e-05 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.362975+0000", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055406810000000004 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363045+0000", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9834999999999998e-05 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363151+0000", 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105624 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.805 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192419 7.6 7:61106e67:::benchmark_data_smithi038_77863_object74151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.33521353499998, 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93476213799999996, 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.806 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192419 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.807 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431360+0000", 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431361+0000", 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1340000000000001e-06 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.808 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431365+0000", 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3619999999999999e-06 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431375+0000", 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0117e-05 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431381+0000", 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0929999999999999e-06 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.809 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431438+0000", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6604000000000002e-05 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.431447+0000", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6519999999999995e-06 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357620+0000", 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92617308499999995 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.357764+0000", 2024-03-04T16:14:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014397800000000001 2024-03-04T16:14:59.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.358209+0000", 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000445043 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365924+0000", 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077151900000000002 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.812 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365936+0000", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.225e-05 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.365979+0000", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2342000000000002e-05 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366027+0000", 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8834e-05 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366122+0000", 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4454000000000001e-05 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192427 7.6 7:63454d35:::benchmark_data_smithi038_77863_object74159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.32747805999998, 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93643149199999998, 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.814 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192427 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.815 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439095+0000", 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439097+0000", 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5549999999999999e-06 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439101+0000", 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1609999999999996e-06 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439109+0000", 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8979999999999996e-06 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439115+0000", 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3980000000000004e-06 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439176+0000", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1017999999999998e-05 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.439183+0000", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4690000000000002e-06 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363184+0000", 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92400118200000003 2024-03-04T16:14:59.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363330+0000", 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146102 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.819 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.363445+0000", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115333 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371572+0000", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081264449999999995 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371613+0000", 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1582000000000003e-05 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375274+0000", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036606170000000001 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375427+0000", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015290900000000001 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375527+0000", 2024-03-04T16:14:59.821 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9822999999999996e-05 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192435 7.6 7:64907183:::benchmark_data_smithi038_77863_object74167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.32132086299998, 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93044704600000006, 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.822 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192435 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445252+0000", 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.823 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445254+0000", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.525e-06 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445258+0000", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4139999999999996e-06 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445266+0000", 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6179999999999997e-06 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445272+0000", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1040000000000003e-06 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445342+0000", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.9524999999999996e-05 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.445350+0000", 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9719999999999992e-06 2024-03-04T16:14:59.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366160+0000", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92081084300000005 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366262+0000", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010210200000000001 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.366364+0000", 2024-03-04T16:14:59.826 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000101168 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371648+0000", 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052841750000000003 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375566+0000", 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039183600000000001 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.827 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375578+0000", 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1749999999999999e-05 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375617+0000", 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9529e-05 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375699+0000", 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1960999999999997e-05 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192455 7.6 7:6a9eb3fa:::benchmark_data_smithi038_77863_object74187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.29901991499997, 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92415198300000001, 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192455 2024-03-04T16:14:59.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467553+0000", 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467555+0000", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6559999999999999e-06 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467560+0000", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5929999999999999e-06 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467568+0000", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3960000000000006e-06 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.831 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467574+0000", 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6559999999999997e-06 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467628+0000", 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4486e-05 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.467646+0000", 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8278e-05 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.832 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.368906+0000", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.901259433 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.369025+0000", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000119533 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.369130+0000", 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105062 2024-03-04T16:14:59.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371681+0000", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025508369999999998 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375706+0000", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040243689999999999 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391530+0000", 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.015824181 2024-03-04T16:14:59.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391593+0000", 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3601000000000003e-05 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391705+0000", 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000111902 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192473 7.6 7:66559066:::benchmark_data_smithi038_77863_object74205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.270798404, 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.896142141, 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192473 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495775+0000", 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.837 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495777+0000", 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3609999999999999e-06 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495785+0000", 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1589999999999993e-06 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.838 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495804+0000", 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8278e-05 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495813+0000", 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1130000000000007e-06 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495882+0000", 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8935000000000003e-05 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.839 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.495892+0000", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9750000000000002e-06 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371709+0000", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87581735599999999 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371819+0000", 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109969 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.371926+0000", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010731 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375711+0000", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037846120000000001 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383485+0000", 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0077737589999999999 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391753+0000", 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00826817 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391819+0000", 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6058999999999999e-05 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.842 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391917+0000", 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8084999999999993e-05 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192480 7.6 7:623fa8f4:::benchmark_data_smithi038_77863_object74212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.259591661, 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88821274100000003, 2024-03-04T16:14:59.843 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192480 2024-03-04T16:14:59.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506982+0000", 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.846 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506983+0000", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.697e-06 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.506989+0000", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507003+0000", 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3961999999999999e-05 2024-03-04T16:14:59.847 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507011+0000", 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2239999999999995e-06 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507066+0000", 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4805999999999997e-05 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.848 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.507074+0000", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4819999999999994e-06 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375735+0000", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86866097200000003 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375842+0000", 2024-03-04T16:14:59.849 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010695499999999999 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.375959+0000", 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011725799999999999 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383524+0000", 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075642280000000001 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.850 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.394497+0000", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010973696 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395111+0000", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061341800000000001 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395156+0000", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5158999999999999e-05 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.851 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395194+0000", 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8393000000000003e-05 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192483 7.6 7:6e5767a6:::benchmark_data_smithi038_77863_object74215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.25583238399997, 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88546532899999997, 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.852 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192483 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510741+0000", 2024-03-04T16:14:59.853 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510743+0000", 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.81e-06 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510749+0000", 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5459999999999997e-06 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.854 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510764+0000", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4452e-05 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510773+0000", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4059999999999994e-06 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510832+0000", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8909999999999997e-05 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.855 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.510843+0000", 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1097999999999999e-05 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380203+0000", 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86935975700000001 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380333+0000", 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012970200000000001 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.856 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.380440+0000", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107753 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383563+0000", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031223760000000001 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395202+0000", 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011639534999999999 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.857 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396095+0000", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00089278899999999995 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396151+0000", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6094000000000002e-05 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396206+0000", 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5101e-05 2024-03-04T16:14:59.858 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192486 7.6 7:62008682:::benchmark_data_smithi038_77863_object74218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.252233236, 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88626451299999998, 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.859 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192486 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514340+0000", 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.860 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514341+0000", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.443e-06 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514347+0000", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3319999999999999e-06 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514362+0000", 2024-03-04T16:14:59.861 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4824000000000001e-05 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514369+0000", 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7849999999999992e-06 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514431+0000", 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1822999999999994e-05 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.862 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.514440+0000", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6389999999999998e-06 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383592+0000", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86915199099999996 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383701+0000", 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109064 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.863 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.383806+0000", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010532299999999999 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391470+0000", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076637140000000003 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395208+0000", 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003738416 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.864 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400506+0000", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0052977520000000002 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400558+0000", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1894999999999997e-05 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.400604+0000", 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6511999999999998e-05 2024-03-04T16:14:59.865 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192499 7.6 7:653bbb72:::benchmark_data_smithi038_77863_object74231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.23727799599999, 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.871864572, 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.866 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192499 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529295+0000", 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.867 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529297+0000", 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.846e-06 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529303+0000", 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6459999999999998e-06 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529316+0000", 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2824e-05 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.868 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529323+0000", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9170000000000006e-06 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529385+0000", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.1257e-05 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.529396+0000", 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1481000000000001e-05 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.869 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386219+0000", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.856823102 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386368+0000", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014870500000000001 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.386471+0000", 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000102537 2024-03-04T16:14:59.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.394461+0000", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0079902649999999999 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395213+0000", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00075248199999999996 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401073+0000", 2024-03-04T16:14:59.871 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005859314 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401122+0000", 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9418e-05 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.401160+0000", 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7778000000000003e-05 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.872 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192505 7.6 7:618cc37a:::benchmark_data_smithi038_77863_object74237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.21731808200002, 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.85606340400000003, 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.873 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192505 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.875 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549255+0000", 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549257+0000", 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1679999999999998e-06 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549265+0000", 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1940000000000001e-06 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.876 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549280+0000", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5509e-05 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549288+0000", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6079999999999998e-06 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549353+0000", 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5479000000000001e-05 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.877 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.549366+0000", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2652e-05 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.388847+0000", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.83948117300000002 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.388967+0000", 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120574 2024-03-04T16:14:59.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.389076+0000", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108552 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395055+0000", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059790970000000001 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395218+0000", 2024-03-04T16:14:59.879 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016246300000000001 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405206+0000", 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099885349999999998 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405266+0000", 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0186999999999998e-05 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.880 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405319+0000", 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2213000000000003e-05 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.14844.0:192534 7.6 7:60f993e4:::benchmark_data_smithi038_77863_object74266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 437.00572376100001, 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.64461839600000004, 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.881 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.14844", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 192534 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760849+0000", 2024-03-04T16:14:59.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760852+0000", 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4669999999999998e-06 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760859+0000", 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3039999999999999e-06 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760873+0000", 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4093999999999999e-05 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760881+0000", 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7060000000000004e-06 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760933+0000", 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2343999999999999e-05 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:42.760941+0000", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8199999999999997e-06 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.391958+0000", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.63101682100000001 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.392074+0000", 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115573 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.392184+0000", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011017499999999999 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.395093+0000", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029088970000000001 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.396983+0000", 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018902109999999999 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405359+0000", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0083764100000000008 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405427+0000", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7448999999999997e-05 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:07:43.405468+0000", 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1124999999999997e-05 2024-03-04T16:14:59.887 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:14:59.889 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:14:59.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114774 8.6 8:694af726:::benchmark_data_smithi038_107057_object114773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 271.53666318000001, 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1166456839999999, 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114774 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.890 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229912+0000", 2024-03-04T16:14:59.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2620000000000001e-06 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229973+0000", 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0553000000000001e-05 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229996+0000", 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2646999999999998e-05 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230005+0000", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4769999999999992e-06 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230092+0000", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6667000000000002e-05 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230253+0000", 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016137700000000001 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,7", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230416+0000", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016295200000000001 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243867+0000", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013450632000000001 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243882+0000", 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5690999999999999e-05 2024-03-04T16:14:59.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232356+0000", 2024-03-04T16:14:59.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98847337300000004 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232375+0000", 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9768000000000001e-05 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.330696+0000", 2024-03-04T16:14:59.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.098320972 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331886+0000", 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011898729999999999 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346472+0000", 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014585653000000001 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346556+0000", 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3787000000000003e-05 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114798 8.6 8:6e193aa2:::benchmark_data_smithi038_107057_object114797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 270.31445539499998, 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.89634460400000004, 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.898 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114798 2024-03-04T16:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452121+0000", 2024-03-04T16:14:59.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.022e-06 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452154+0000", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2851000000000001e-05 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452177+0000", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2880000000000001e-05 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452185+0000", 2024-03-04T16:14:59.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8149999999999995e-06 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452277+0000", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1267999999999994e-05 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452292+0000", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5811000000000002e-05 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332203+0000", 2024-03-04T16:14:59.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87991063700000005 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332343+0000", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013942199999999999 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332826+0000", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048315199999999998 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346595+0000", 2024-03-04T16:14:59.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013768927 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346632+0000", 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7630999999999999e-05 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346680+0000", 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7589999999999997e-05 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348078+0000", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013984640000000001 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348149+0000", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0673000000000006e-05 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348462+0000", 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031346100000000001 2024-03-04T16:14:59.905 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114804 8.6 8:642351e4:::benchmark_data_smithi038_107057_object114803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 270.29283041000002, 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87523897799999995, 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.906 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114804 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473746+0000", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8600000000000001e-06 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473780+0000", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4461000000000001e-05 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473798+0000", 2024-03-04T16:14:59.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8297000000000001e-05 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473809+0000", 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473890+0000", 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0513999999999995e-05 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473913+0000", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3564e-05 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343022+0000", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86910901200000001 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343537+0000", 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051479200000000003 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343752+0000", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000215195 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346685+0000", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002932933 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.347988+0000", 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013024550000000001 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348035+0000", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6859e-05 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348780+0000", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074556000000000004 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348845+0000", 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4453000000000001e-05 2024-03-04T16:14:59.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348982+0000", 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013713899999999999 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:207574 8.6 8:63054e5a:::benchmark_data_smithi038_107057_object85939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 173.264687536, 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.183928063, 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:14:59.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 207574 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:14:59.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501887+0000", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.246e-06 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503000+0000", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001113459 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:14:59.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503016+0000", 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5690000000000001e-05 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503024+0000", 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3299999999999999e-06 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507462+0000", 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044379500000000004 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507549+0000", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6785000000000003e-05 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507678+0000", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012845599999999999 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.509750+0000", 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020719219999999999 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.510463+0000", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000713779 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685583+0000", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17511929700000001 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685760+0000", 2024-03-04T16:14:59.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177682 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685814+0000", 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3467000000000002e-05 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:14:59.919 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:14:59.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:00.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:59 smithi176 ceph-mon[33619]: 6.1f deep-scrub starts 2024-03-04T16:15:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:59 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:15:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:14:59 smithi176 ceph-mon[33619]: 6.1f deep-scrub ok 2024-03-04T16:15:00.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:00.173 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:00.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:00.323 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:00.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:00.460 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:00.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:00.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:15:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:00.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:00.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:00.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:00.872 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:00.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:00.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: pgmap v549: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 9.4 deep-scrub starts 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: osdmap e278: 8 total, 8 up, 6 in 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:15:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 3.b deep-scrub starts 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[29779]: 3.b deep-scrub ok 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: pgmap v549: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 9.4 deep-scrub starts 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: osdmap e278: 8 total, 8 up, 6 in 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:15:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 3.b deep-scrub starts 2024-03-04T16:15:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:00 smithi038 ceph-mon[38113]: 3.b deep-scrub ok 2024-03-04T16:15:01.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:15:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:15:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:15:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: pgmap v549: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 9.4 deep-scrub starts 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: osdmap e278: 8 total, 8 up, 6 in 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 3.b deep-scrub starts 2024-03-04T16:15:01.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:00 smithi176 ceph-mon[33619]: 3.b deep-scrub ok 2024-03-04T16:15:01.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:01.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:01.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:01.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:01.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:01.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:01.460 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:01.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:01.602 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:01.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.728 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:01.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:01.829 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:01.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:01.964 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:01.989 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:01.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:01 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:15:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:01 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:15:01.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:01 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:15:01.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:01 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:15:02.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:01 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:15:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:01 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:15:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:02.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:02.224 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.258 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:02.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:02.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:02.384 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:02.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.522 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:02.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:02.623 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:02.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:02.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:02.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:02.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:02.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:02.924 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:02.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:02.987 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:15:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:15:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-mon[33619]: pgmap v551: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:15:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:15:02.988 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:15:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:15:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:15:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[29779]: pgmap v551: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:15:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:15:02.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:15:02.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:15:02.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:15:02.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[38113]: pgmap v551: 148 pgs: 148 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-04T16:15:02.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:15:02.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:02 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:15:03.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:03.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:03.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:03.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:03.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:03.359 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:02 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:03.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:03.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:03.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.493 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:03.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:03.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:03.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:03.680 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:03.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:03.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:03.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:15:03.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:03.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:03.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[29779]: 9.f scrub starts 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[29779]: 5.b deep-scrub starts 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[29779]: 5.b deep-scrub ok 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[38113]: 9.f scrub starts 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[38113]: 5.b deep-scrub starts 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[38113]: 5.b deep-scrub ok 2024-03-04T16:15:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:04.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:04.010 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:04.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:04.078 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:03 smithi176 ceph-mon[33619]: 9.f scrub starts 2024-03-04T16:15:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:03 smithi176 ceph-mon[33619]: 5.b deep-scrub starts 2024-03-04T16:15:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:03 smithi176 ceph-mon[33619]: 5.b deep-scrub ok 2024-03-04T16:15:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:04.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:04.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:04.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:04.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:04.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:04.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:04.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:04.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:04.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:04.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:04.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:04.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:04.857 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:04.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:04.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[38113]: pgmap v552: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-04T16:15:04.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:15:04.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:15:04.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[38113]: 9.a scrub starts 2024-03-04T16:15:04.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[29779]: pgmap v552: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-04T16:15:04.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:15:04.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:15:04.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:04 smithi038 ceph-mon[29779]: 9.a scrub starts 2024-03-04T16:15:04.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:05.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:05.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:05.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:04 smithi176 ceph-mon[33619]: pgmap v552: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.8 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-04T16:15:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:04 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:15:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:04 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:15:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:04 smithi176 ceph-mon[33619]: 9.a scrub starts 2024-03-04T16:15:05.144 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:05.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:05.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:05.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:05.421 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:05.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:05.663 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.699 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:05.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:05.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:05.832 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:05.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[29779]: 9.f scrub ok 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[29779]: 9.6 scrub starts 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[38113]: 9.f scrub ok 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:15:05.952 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:05 smithi038 ceph-mon[38113]: 9.6 scrub starts 2024-03-04T16:15:05.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:05.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:06.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:05 smithi176 ceph-mon[33619]: 9.f scrub ok 2024-03-04T16:15:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:05 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:15:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:05 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:15:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:05 smithi176 ceph-mon[33619]: 9.6 scrub starts 2024-03-04T16:15:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:06.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:06.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:06.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.396 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:06.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:06.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:06.539 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:06.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:06.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:06.667 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:15:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.682 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:06.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:06.708 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:15:06.750 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:15:06.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:06.795 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:15:06.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:06.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[29779]: 9.a scrub ok 2024-03-04T16:15:06.836 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[29779]: pgmap v553: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:15:06.836 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2733490062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:06.836 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[38113]: 9.a scrub ok 2024-03-04T16:15:06.836 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[38113]: pgmap v553: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:15:06.836 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2733490062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:06.836 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:15:06.877 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:15:06.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:15:06.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:06.962 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:15:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:06.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:06.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:06.964 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:15:06.965 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:15:07.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:07.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:06 smithi176 ceph-mon[33619]: 9.a scrub ok 2024-03-04T16:15:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:06 smithi176 ceph-mon[33619]: pgmap v553: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:15:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2733490062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:07.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:07.238 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:07.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:07.382 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:07.484 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:07.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:07.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.653 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:07.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:07.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:07.784 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:07.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 9.6 scrub ok 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:15:07.907 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 9.6 scrub ok 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:15:07.908 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:07 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:15:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:07.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:07.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:08.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:08.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 9.6 scrub ok 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:15:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:15:08.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:07 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:15:08.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:08.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:08.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.344 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:08.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:08.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:08.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:08.520 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:08.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:08.622 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:08.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:08.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: pgmap v554: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:15:08.870 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: pgmap v554: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:15:08.871 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:08 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:15:08.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:08.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:08.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:09.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:09.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: pgmap v554: 148 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 146 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:15:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:08 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:15:09.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.182 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:09.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:09.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:09.311 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:09.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:09.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:09.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:09.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.590 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:09.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:09.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:09.721 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:09.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:09.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:09.864 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:09.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:09.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.000 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:10.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:10.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:09 smithi176 ceph-mon[33619]: 9.7 deep-scrub starts 2024-03-04T16:15:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:09 smithi176 ceph-mon[33619]: 9.0 scrub starts 2024-03-04T16:15:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.144 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:10.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:10.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:09 smithi038 ceph-mon[38113]: 9.7 deep-scrub starts 2024-03-04T16:15:10.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:09 smithi038 ceph-mon[38113]: 9.0 scrub starts 2024-03-04T16:15:10.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:09 smithi038 ceph-mon[29779]: 9.7 deep-scrub starts 2024-03-04T16:15:10.145 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:09 smithi038 ceph-mon[29779]: 9.0 scrub starts 2024-03-04T16:15:10.246 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:10.275 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:10.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:10.420 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:10.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:10.553 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:10.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:15:10.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.698 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:10.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:10.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:10.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:10.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:10.964 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:10.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:10.989 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:11.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:11.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:11.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:11.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:11.236 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:11.311 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 1 pgp_num to 1 2024-03-04T16:15:11.315 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:15:11.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:11.387 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:11.489 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.522 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:11.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: 9.0 scrub ok 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:15:11.565 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:15:11.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:11.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:15:11.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:15:11.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: 9.0 scrub ok 2024-03-04T16:15:11.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:15:11.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:15:11.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:15:11.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: pgmap v555: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: 9.0 scrub ok 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:15:11.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:15:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:15:11.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-04T16:15:11.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:11.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:11.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:11.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:11.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:11.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:11.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:12.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:12.080 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:12.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.219 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:12.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:12.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:12.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:12.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:12.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 6.15 deep-scrub starts 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: 6.15 deep-scrub ok 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:15:12.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[29779]: osdmap e279: 8 total, 8 up, 6 in 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 6.15 deep-scrub starts 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: 6.15 deep-scrub ok 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:12.566 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:15:12.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:15:12.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:12 smithi038 ceph-mon[38113]: osdmap e279: 8 total, 8 up, 6 in 2024-03-04T16:15:12.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:15:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:15:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:15:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:15:12.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 6.15 deep-scrub starts 2024-03-04T16:15:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: 6.15 deep-scrub ok 2024-03-04T16:15:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1557009337' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:15:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2345111285' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-04T16:15:12.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-mon[33619]: osdmap e279: 8 total, 8 up, 6 in 2024-03-04T16:15:12.619 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:12.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:12.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:12.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:12.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:12.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:12.893 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:12.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:12.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:13.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:13.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:13.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:13.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:13.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:13.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:15:13.374 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:15:13.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.439 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:13.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:13 smithi038 ceph-mon[29779]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:13.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:13 smithi038 ceph-mon[38113]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:13.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:13.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:13.566 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:13 smithi176 ceph-mon[33619]: pgmap v557: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:15:13.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:13.703 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:13.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:13.838 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:13.939 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:13.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:14.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:14.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:14.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:14.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:14.347 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:14.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:14.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:14.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:15:14.519 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:14.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:14.560 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:14.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:14.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:14.651 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:14.753 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:14.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:14.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:14.916 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:14.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:15.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.054 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:15.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:15.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:15.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:15.289 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:15.319 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:15.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:15.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:15:15.453 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:15.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:15.571 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:15 smithi038 ceph-mon[29779]: pgmap v558: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:15:15.571 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:15 smithi038 ceph-mon[38113]: pgmap v558: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:15:15.589 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:15.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:15 smithi176 ceph-mon[33619]: pgmap v558: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.2 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:15:15.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:15.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:15.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:15.857 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:15.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:15.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:15.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:16.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:16.127 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:16.142 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:15:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.151 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:16.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:16.188 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:15:16.230 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:15:16.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:16.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:15:16.282 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:16.304 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:15:16.342 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:15:16.383 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:15:16.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:16.423 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:15:16.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:16.430 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:15:16.431 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:15:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:16 smithi038 ceph-mon[29779]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:16.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2096480039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:16.449 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:16 smithi038 ceph-mon[38113]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:16.449 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2096480039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:16.526 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:16.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:16 smithi176 ceph-mon[33619]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2096480039' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:16.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:16.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:16.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:16.834 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:16.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:16.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:16.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:17.071 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:17.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:17.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:17.235 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:17.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:17.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:17.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.510 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:17.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:17.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.645 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:17.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:17.746 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:17.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:17.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:17.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:17.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:17.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:18.025 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:18.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:18.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:18.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:18.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:18.328 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:18.330 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:18.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:18.468 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:18.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:18.598 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:18.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:18.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.729 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:18.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:18.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:18 smithi038 ceph-mon[29779]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:18.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:18 smithi038 ceph-mon[38113]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:18.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:18 smithi176 ceph-mon[33619]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 6.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:18.862 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:18.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:18.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:18.999 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:19.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:15:19.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:15:19.047 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-04T16:15:19.047 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-04T16:15:19.047 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-04T16:15:19.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-04T16:15:19.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.145 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:19.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:19.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:19.284 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:19.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:19.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:19.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:19.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:15:19.554 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:19.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:19.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:19.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:19.831 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:19.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:19.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:19.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:20.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:20.111 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:20.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:20.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:20.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:20.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:20.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:20.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:20.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:20.646 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:20.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:20.785 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:20.866 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:20 smithi038 ceph-mon[29779]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s 2024-03-04T16:15:20.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:20 smithi038 ceph-mon[38113]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s 2024-03-04T16:15:20.887 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:20.919 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:20.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:21.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.059 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:21.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:20 smithi176 ceph-mon[33619]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 939 op/s 2024-03-04T16:15:21.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.196 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:21.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:21.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:21.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:21.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:21.465 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:21.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:21.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:21.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.738 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:21.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:21.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:21.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:21.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:21.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:22.008 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:22.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.152 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:22.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:22.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:22.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:22.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:22.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:22.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:22.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:22.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:22.697 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:22.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:22.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:22 smithi038 ceph-mon[29779]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 918 op/s 2024-03-04T16:15:22.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:22 smithi038 ceph-mon[38113]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 918 op/s 2024-03-04T16:15:22.876 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:15:22.876 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:15:22.877 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:15:22.877 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:15:22.877 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:15:22.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:22.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:22.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:23.007 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:22 smithi176 ceph-mon[33619]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1000 B/s rd, 0 B/s wr, 918 op/s 2024-03-04T16:15:23.067 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:23.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:23.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:23.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:23.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:23.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:23.375 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:23.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:23.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:23.626 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:23.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:23.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.673 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:23.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:23.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:15:23.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:15:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.825 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:23.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:23.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:23.959 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:23.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4236298113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[29779]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:23.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:15:23.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:15:23.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4236298113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:23.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:23 smithi038 ceph-mon[38113]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:24.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:24.092 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4236298113' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:23 smithi176 ceph-mon[33619]: from='client.25432 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:15:24.174 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:24.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.248 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:24.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:24.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:24.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.388 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:24.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:24.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:24.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:24.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:24.674 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:24.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:24.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:24.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:24.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:24.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:24.955 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:24.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:25.057 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:25.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:24 smithi176 ceph-mon[33619]: pgmap v563: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:24 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:24 smithi176 ceph-mon[33619]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:15:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:24 smithi176 ceph-mon[33619]: osdmap e280: 8 total, 8 up, 6 in 2024-03-04T16:15:25.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:25.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[29779]: pgmap v563: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[29779]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[29779]: osdmap e280: 8 total, 8 up, 6 in 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[38113]: pgmap v563: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.6 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[38113]: from='client.25432 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:15:25.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:24 smithi038 ceph-mon[38113]: osdmap e280: 8 total, 8 up, 6 in 2024-03-04T16:15:25.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:25.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:25.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:25.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:25.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:25.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:25.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:25.776 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:25.878 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:25.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:25.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:25.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:26.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:26.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:26.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.175 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:26.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:26.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.311 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:26.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:26.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:26.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:26.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:26.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:26.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:26.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:26.814 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:26.856 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:26.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:26.991 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:26.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:27.093 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:27.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.127 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:27.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:27.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.265 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:27.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:27.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:27.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:27.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:27.538 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:27 smithi176 ceph-mon[33619]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:27.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:27.677 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:27 smithi038 ceph-mon[29779]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:27 smithi038 ceph-mon[38113]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:27.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:27.813 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:27.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:27.948 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:27.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:28.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.089 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:28.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:28.190 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.228 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:28.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:28.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:28.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.497 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:28.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:28.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:28.707 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:28.708 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:28.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:28.758 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:28.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:28.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:28.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:28.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:29.031 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:29.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:29.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:29.275 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:29.307 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:29.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:29.440 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:29.542 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:29.579 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:29 smithi176 ceph-mon[33619]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:29.681 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:29.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:29 smithi038 ceph-mon[29779]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:29.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:29 smithi038 ceph-mon[38113]: pgmap v566: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 904 op/s 2024-03-04T16:15:29.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:29.850 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:29.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:29.995 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:29.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:30.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:30.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:30.233 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.269 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:30.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:30.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:30.406 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:30.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:30.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.547 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:30.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:30.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:30.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.682 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:30.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:30.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:15:30.784 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.821 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:30.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:30.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:30.960 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:30.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:31.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.097 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:31.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:31.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:31.239 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:31.341 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:31.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:31.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:31 smithi038 ceph-mon[29779]: pgmap v567: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:31 smithi038 ceph-mon[38113]: pgmap v567: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:31.505 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:31.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:31 smithi176 ceph-mon[33619]: pgmap v567: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:31.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:31.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:31.779 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:31.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:31.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:31.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:32.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.062 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:32.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:32.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:32.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:32.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:32.351 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:32.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:32.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:32.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:32.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:32.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:32.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:32.770 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:32.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:32.905 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:32.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:33.007 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:33.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:33.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:33.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:33.183 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:33.285 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:33.320 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:33.324 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:33.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.462 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:33.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:33.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:33.601 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:33.605 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:33.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:33 smithi038 ceph-mon[29779]: pgmap v568: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:33.606 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:33 smithi038 ceph-mon[38113]: pgmap v568: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:33 smithi176 ceph-mon[33619]: pgmap v568: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:15:33.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:33.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:33.893 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:33.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:33.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:34.067 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:34.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.162 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:34.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:34.295 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:34.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:34.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:34 smithi038 ceph-mon[29779]: pgmap v569: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:15:34.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:34.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:34 smithi038 ceph-mon[38113]: pgmap v569: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:15:34.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:34.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:34 smithi176 ceph-mon[33619]: pgmap v569: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:15:34.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:34.710 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:34.812 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:34.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:34.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:34.982 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:34.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:35.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:35.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:35.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:35.144 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:15:35.183 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:15:35.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:35.223 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:15:35.246 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:15:35.247 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "" 2024-03-04T16:15:35.247 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:15:35.249 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:35.266 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:15:35.310 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:15:35.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:35.354 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:15:35.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:35.391 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:15:35.392 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2434585610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:35.392 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2434585610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:35.430 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:15:35.436 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:15:35.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.527 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:35.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2434585610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:35.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:35.657 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:35.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:35.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:35.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:35.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:36.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.067 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:36.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:36.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:36.206 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:36.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:36.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:36.340 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:36.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.486 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:36.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:36.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:36 smithi176 ceph-mon[33619]: pgmap v570: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 869 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:36.619 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:36.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:36.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:36 smithi038 ceph-mon[29779]: pgmap v570: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 869 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:36.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:36 smithi038 ceph-mon[38113]: pgmap v570: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 869 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:36.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:36.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:36.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:36.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:36.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:37.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.031 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:37.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:37.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.185 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:37.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:37.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:37.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.323 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:37.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:37.424 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.464 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:37.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:37.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:37.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:37.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:37.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:37.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:37.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:37.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:37.971 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:38.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:38.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:38.140 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:38.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:38.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:38.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:38.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:38.427 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:15:38.427 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:15:38.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:38.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:38.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:38.840 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[29779]: pgmap v571: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[38113]: pgmap v571: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:38.892 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:15:38.893 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:38 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:15:38.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:38.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:38.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:39.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:38 smithi176 ceph-mon[33619]: pgmap v571: 148 pgs: 148 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-04T16:15:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:38 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:15:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:38 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:15:39.114 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:39.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:39.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:39.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:39.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:39.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:39.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:39.526 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:39.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.668 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:39.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:39.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:39.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.811 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:39.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:39.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:39.935 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:39.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:39.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:39 smithi038 ceph-mon[29779]: osdmap e281: 8 total, 8 up, 6 in 2024-03-04T16:15:39.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:39 smithi038 ceph-mon[38113]: osdmap e281: 8 total, 8 up, 6 in 2024-03-04T16:15:40.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:40.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:40.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:39 smithi176 ceph-mon[33619]: osdmap e281: 8 total, 8 up, 6 in 2024-03-04T16:15:40.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:15:40.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:40.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:40.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.349 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:40.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:40.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:40.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:40.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:40.620 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:40.666 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:40 smithi038 ceph-mon[29779]: pgmap v573: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:40.667 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:40 smithi038 ceph-mon[38113]: pgmap v573: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:40.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:40.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:40.819 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:40.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:40 smithi176 ceph-mon[33619]: pgmap v573: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:40.880 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:40.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:41.014 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:41.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:41.161 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:41.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:41.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:41.296 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:41.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:41.432 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:41.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:41.568 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:41.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:41.694 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:15:41.709 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:41.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:41.847 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:41.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:41.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:41.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:41.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/372467636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:41.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:41 smithi038 ceph-mon[29779]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/372467636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:41 smithi038 ceph-mon[38113]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:42.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/372467636' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:41 smithi176 ceph-mon[33619]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:15:42.115 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:42.217 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:42.249 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:42.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:42.385 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:42.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:15:42.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:42.584 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:42.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:42.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:42.715 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:42.789 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[29779]: pgmap v574: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:42.789 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:42.789 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[29779]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:15:42.789 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[29779]: osdmap e282: 8 total, 8 up, 6 in 2024-03-04T16:15:42.790 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[38113]: pgmap v574: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:42.790 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:42.790 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[38113]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:15:42.790 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:42 smithi038 ceph-mon[38113]: osdmap e282: 8 total, 8 up, 6 in 2024-03-04T16:15:42.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:42.850 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:42.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:42.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:42.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:43.028 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:42 smithi176 ceph-mon[33619]: pgmap v574: 148 pgs: 148 active+clean; 4.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.51k op/s 2024-03-04T16:15:43.028 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:42 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:15:43.028 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:42 smithi176 ceph-mon[33619]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:15:43.028 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:42 smithi176 ceph-mon[33619]: osdmap e282: 8 total, 8 up, 6 in 2024-03-04T16:15:43.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:43.113 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:43.215 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:43.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:43.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:43.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:43.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:43.436 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:15:43.436 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:15:43.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:43.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:43.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:43.663 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:43.675 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:43.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:43.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:43.942 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:15:43.942 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:15:43.942 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-04T16:15:43.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:15:43.947 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:43.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:43.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:15:43.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:44.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:44.092 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:15:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:15:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:15:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:15:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:44.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:44.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:44.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:44.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:44.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:44.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:44.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:15:44.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:44.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:44.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:44.678 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:44.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:44.759 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:44.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[29779]: pgmap v576: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[38113]: pgmap v576: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-04T16:15:44.825 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:44 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:15:44.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:44.902 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:44.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:45.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:45.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:44 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:15:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:44 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:15:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:44 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:15:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:44 smithi176 ceph-mon[33619]: pgmap v576: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.61k op/s 2024-03-04T16:15:45.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:44 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:15:45.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:45.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:45.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:45.302 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:45.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.437 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:45.525 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:45.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:45.570 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:45.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.708 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:45.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:45.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:15:45.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:45.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:45.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:45.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:45.989 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:15:45.989 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:15:45.989 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:15:45.989 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:15:45.990 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:15:45.990 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:15:45.990 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:15:45.990 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:45 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:15:46.031 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:46.077 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:46.106 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:45 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:15:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:45 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:15:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:45 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:15:46.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:45 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:15:46.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:46.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:46.347 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.383 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:46.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:46.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:46.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:46.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.669 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:46.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:46.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:46.808 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:46.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:46.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:46.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:47.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:47.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:47.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:47.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:47.220 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:47.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:47.355 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:47.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:15:47.511 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:15:47.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:15:47.587 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: pgmap v577: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 9.4 scrub starts 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:15:47.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3517404471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: pgmap v577: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 9.4 scrub starts 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:15:47.589 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3517404471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:47.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:47.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:15:47.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:15:47.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:15:47.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:15:47.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:15:47.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:15:47.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:15:47.594 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: pgmap v577: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 9.4 scrub starts 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:15:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:15:47.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3517404471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:15:47.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:47.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:47.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:47.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:47.903 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:47.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:48.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:48.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:48.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:48.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:48.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:48.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:48.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:48.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 9.7 scrub starts 2024-03-04T16:15:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:15:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:15:48.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[29779]: 9.4 scrub ok 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 9.7 scrub starts 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:15:48.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:48 smithi038 ceph-mon[38113]: 9.4 scrub ok 2024-03-04T16:15:48.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 9.7 scrub starts 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:15:48.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:48 smithi176 ceph-mon[33619]: 9.4 scrub ok 2024-03-04T16:15:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:48.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:48.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:48.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.761 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:48.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:48.862 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:48.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:48.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:48.998 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:49.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:49.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:49.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:49.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:49.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:49.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:49.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:49.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:49.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: pgmap v578: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 9.7 scrub ok 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:15:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:49 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:15:49.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:49.724 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: pgmap v578: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 9.7 scrub ok 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:15:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: pgmap v578: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 9.7 scrub ok 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:15:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:49 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:15:49.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:49.865 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:49.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:49.967 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:50.002 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:50.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:50.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:50.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:50.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:50.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:50.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.552 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:50.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:15:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:15:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:15:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:15:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:15:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:15:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:15:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:15:50.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:50 smithi176 ceph-mon[33619]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:50.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:50.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:15:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[29779]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:15:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:50 smithi038 ceph-mon[38113]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:50.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:50.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:50.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:50.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:50.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:51.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:51.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:51.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:51.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:51.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:51.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:51.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:51.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:51.504 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:51.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.648 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:51.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:51.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.788 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:51.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:51.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:51.932 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:51.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:52.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:52.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:52.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:52.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:52.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:52.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:52.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:52.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:52.589 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:52.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:52.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:52.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:52.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:52.899 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:52.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[29779]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:15:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:15:52.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:15:52.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:52 smithi038 ceph-mon[38113]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:53.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.036 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:53.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:53.038 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:52 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:15:53.038 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:52 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:15:53.038 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:52 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:15:53.038 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:52 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:15:53.038 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:52 smithi176 ceph-mon[33619]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.42k op/s 2024-03-04T16:15:53.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:53.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:53.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:53.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:53.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:15:53.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.445 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:53.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:53.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:53.555 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:15:53.574 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:53.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:15:53.641 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:15:53.641 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:15:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:15:53.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.712 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:53.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:53.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:53.850 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:53.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:53.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:53.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:15:53.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:15:53.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:53.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:53.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:53.995 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:53.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:54.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:15:54.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:15:54.127 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:54.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:54.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:15:54.265 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:54.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:54.398 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:54.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.544 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:54.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.692 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:54.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.845 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:54.901 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[29779]: pgmap v581: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-04T16:15:54.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:15:54.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:15:54.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:54.904 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[38113]: pgmap v581: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-04T16:15:54.904 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:15:54.904 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:15:54.904 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:54.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:54.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:54.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:55.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 282, 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:15:55.105 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:15:41.639406+0000", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:10:28.206965+0000", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5804032, 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:15:55.106 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 9, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.107 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:15:55.108 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "279", 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.109 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.110 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.111 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.112 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:15:55.113 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:15:55.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:15:55.115 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.116 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.117 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.118 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6500000953674316, 2024-03-04T16:15:55.119 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6500000953674316, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.62999999523162842, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.6699999570846558, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.6699999570846558, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:15:55.120 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:15:55.121 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.122 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.124 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.125 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.75, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.75, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.309999942779541, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.309999942779541, 2024-03-04T16:15:55.126 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.127 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.128 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.129 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.130 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.131 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5099999904632568, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5099999904632568, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:15:55.132 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:15:55.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.134 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.135 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.136 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.137 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.138 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:15:55.139 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:15:55.139 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:15:55.139 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:15:55.139 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:15:55.139 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:54 smithi176 ceph-mon[33619]: pgmap v581: 148 pgs: 148 active+clean; 3.0 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-03-04T16:15:55.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:54 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:15:55.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:54 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:15:55.140 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.140 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.141 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.144 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.145 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:15:55.146 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 9, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_2", 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:12:56.904974+0000", 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:15:55.147 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:15:55.148 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "247", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:15:55.149 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:15:55.150 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:15:55.151 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:15:55.152 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.153 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 242, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:15:55.154 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.155 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.156 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:15:55.157 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 242, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.158 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.160 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 140, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.162 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:15:55.163 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.164 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:15:55.165 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.166 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 242, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.167 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.168 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.169 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:15:55.171 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.172 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 209, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:15:55.173 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.176 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 242, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:15:55.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.178 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:15:55.179 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:15:55.180 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:15:55.181 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 190, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.182 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.184 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:15:55.185 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 242, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.186 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.187 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:15:55.188 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:15:55.189 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:14:41.055065+0000", 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.190 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:14:59.306289+0000", 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:15:55.191 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:14:52.088493+0000", 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:15:23.422990+0000", 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.192 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:15:38.175270+0000", 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.193 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:14:44.466808+0000", 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:15:10.454319+0000", 2024-03-04T16:15:55.194 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:14:50.098470+0000", 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-04T16:15:55.195 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.13", 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.1e", 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:15:55.196 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 3 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.197 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:15:55.198 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:15:55.199 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:15:55.200 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:15:55.200 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:15:55.200 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:15:55.200 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:15:55.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.201 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:55.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:55.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:55.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.341 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:55.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:55.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.484 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:55.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:55.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.617 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:55.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:55.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.771 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:55.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:55.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:55.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[38113]: 9.1 scrub starts 2024-03-04T16:15:55.903 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[38113]: 9.e scrub starts 2024-03-04T16:15:55.904 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3866710160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:15:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[29779]: 9.1 scrub starts 2024-03-04T16:15:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[29779]: 9.e scrub starts 2024-03-04T16:15:55.905 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3866710160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:15:55.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:55.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:55.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:56.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:56.056 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:55 smithi176 ceph-mon[33619]: 9.1 scrub starts 2024-03-04T16:15:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:55 smithi176 ceph-mon[33619]: 9.e scrub starts 2024-03-04T16:15:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3866710160' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:15:56.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:15:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.190 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:56.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:15:56.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:56.318 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:15:56.318 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:15:56.318 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:15:56.318 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:15:56.319 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:15:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.332 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:56.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:56.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.467 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:56.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:56.568 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:56.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:56.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.739 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:56.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:56.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:56.876 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:56.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:56.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 9.e scrub ok 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 9.1 scrub ok 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: pgmap v582: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 9.e scrub ok 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 9.1 scrub ok 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: pgmap v582: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:56 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.020 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:57.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 9.e scrub ok 2024-03-04T16:15:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 9.1 scrub ok 2024-03-04T16:15:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:15:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:15:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: pgmap v582: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s 2024-03-04T16:15:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:15:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:15:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:15:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:15:57.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:56 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:15:57.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:57.163 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:57.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:57.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:57.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:57.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:57.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:57.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:57.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:57.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:57.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:57.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:57.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:57.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:15:57.991 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:57.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:15:57.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 9.c scrub starts 2024-03-04T16:15:57.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 9.3 scrub starts 2024-03-04T16:15:57.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:15:57.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 9.c scrub starts 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 9.3 scrub starts 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:15:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:57 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:15:58.093 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 9.c scrub starts 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 9.3 scrub starts 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:15:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:57 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:15:58.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:58.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:15:58.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.262 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:58.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:15:58.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:58.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:58.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.526 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:58.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:58.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:58.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:58.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:58.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:58.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:58.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:58.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:58.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[29779]: 9.2 scrub starts 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[29779]: 9.c scrub ok 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[29779]: 9.3 scrub ok 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[29779]: pgmap v583: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[38113]: 9.2 scrub starts 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[38113]: 9.c scrub ok 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[38113]: 9.3 scrub ok 2024-03-04T16:15:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:58 smithi038 ceph-mon[38113]: pgmap v583: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:58 smithi176 ceph-mon[33619]: 9.2 scrub starts 2024-03-04T16:15:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:58 smithi176 ceph-mon[33619]: 9.c scrub ok 2024-03-04T16:15:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:58 smithi176 ceph-mon[33619]: 9.3 scrub ok 2024-03-04T16:15:59.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:58 smithi176 ceph-mon[33619]: pgmap v583: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:15:59.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:59.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:59.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:15:59.205 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:15:59.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:59.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:59.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:15:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:15:59.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:15:59.625 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:15:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:15:59.727 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 9.2 scrub ok 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 9.d scrub starts 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 9.2 scrub ok 2024-03-04T16:15:59.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 9.d scrub starts 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:15:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:15:59 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:15:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.763 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:59.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:15:59.864 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:15:59.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:15:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:15:59.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:15:59.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:15:59.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:00.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 9.2 scrub ok 2024-03-04T16:16:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 9.d scrub starts 2024-03-04T16:16:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:16:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:16:00.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:16:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:16:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:16:00.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:15:59 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:16:00.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:00.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:00.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:00.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.329 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:00.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:00.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:00.468 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:00.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:00.609 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:00.638 INFO:tasks.thrashosds.thrasher:j is {'epoch': 282, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:15:41.639406+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:10:28.206965+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-04T16:12:56.904974+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '247', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 242, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:14:41.055065+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:14:59.306289+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:14:52.088493+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:15:23.422990+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:15:38.175270+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:14:44.466808+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:15:10.454319+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:14:50.098470+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '5.13', 'mappings': [{'from': 0, 'to': 5}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:16:00.638 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 5.1e 2024-03-04T16:16:00.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd rm-pg-upmap-items 5.1e 2024-03-04T16:16:00.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:00.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:00.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:00.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:00.869 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:00.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:00.936 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:00 smithi038 ceph-mon[29779]: 9.d scrub ok 2024-03-04T16:16:00.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:00 smithi038 ceph-mon[29779]: pgmap v584: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:16:00.937 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:00 smithi038 ceph-mon[38113]: 9.d scrub ok 2024-03-04T16:16:00.937 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:00 smithi038 ceph-mon[38113]: pgmap v584: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:16:00.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:01.014 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:00 smithi176 ceph-mon[33619]: 9.d scrub ok 2024-03-04T16:16:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:00 smithi176 ceph-mon[33619]: pgmap v584: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s 2024-03-04T16:16:01.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:01.153 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:01.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:01.288 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:01.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:01.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:01.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:01.564 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:01.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:01.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:01.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:01.841 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:01.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:01.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:01.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:01.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[29779]: osdmap e283: 8 total, 8 up, 6 in 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:01.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:01 smithi038 ceph-mon[38113]: osdmap e283: 8 total, 8 up, 6 in 2024-03-04T16:16:02.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:02.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:01 smithi176 ceph-mon[33619]: osdmap e283: 8 total, 8 up, 6 in 2024-03-04T16:16:02.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:02.246 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:02.348 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:02.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:02.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:02.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:02.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.651 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:02.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:02.790 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:02.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:02.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:02.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:02.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: pgmap v586: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[29779]: osdmap e284: 8 total, 8 up, 6 in 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: pgmap v586: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:02 smithi038 ceph-mon[38113]: osdmap e284: 8 total, 8 up, 6 in 2024-03-04T16:16:03.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: pgmap v586: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:03.049 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:02 smithi176 ceph-mon[33619]: osdmap e284: 8 total, 8 up, 6 in 2024-03-04T16:16:03.064 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:03.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:03.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.207 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:03.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:03.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:03.342 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:03.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:03.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:03.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:03.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:03.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:03.622 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:03.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:03.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:03.737 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:03.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:03.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:03.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:03.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:03.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:16:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:16:03.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:04.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:04.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:16:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:04.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:04.218 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:04.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:04.353 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:04.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.492 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:04.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:04.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:04.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:04.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:16:04.769 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:04.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:04.902 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:04.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:04.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: pgmap v588: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3738433756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[38113]: from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:16:04.971 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: pgmap v588: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:16:04.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3738433756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:04.972 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:04 smithi038 ceph-mon[29779]: from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:05.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.052 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: pgmap v588: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3738433756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:04 smithi176 ceph-mon[33619]: from='client.35483 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:05.153 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.190 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:05.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:05.317 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:05.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:05.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:05.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:05.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:05.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:05.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:05.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:05.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:05.994 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:05.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:06.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:16:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:05 smithi176 ceph-mon[33619]: osdmap e285: 8 total, 8 up, 6 in 2024-03-04T16:16:06.133 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:06.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:06.224 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[29779]: osdmap e285: 8 total, 8 up, 6 in 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: from='client.35483 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:06.225 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:05 smithi038 ceph-mon[38113]: osdmap e285: 8 total, 8 up, 6 in 2024-03-04T16:16:06.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:06.294 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:06.396 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.430 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:06.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:06.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.567 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:06.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:06.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:06.701 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:06.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:06.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:06.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:06.988 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:06.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:06.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:16:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:16:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[29779]: pgmap v590: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-04T16:16:06.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:16:06.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:16:06.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:06 smithi038 ceph-mon[38113]: pgmap v590: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-04T16:16:07.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:06 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:16:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:06 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:16:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:06 smithi176 ceph-mon[33619]: pgmap v590: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-04T16:16:07.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:07.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.260 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:07.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:07.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:07.397 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:07.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:07.537 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:07.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:07.670 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:07.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:16:07.772 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:07.791 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:16:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.808 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:07.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:07.836 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:16:07.877 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[29779]: 9.8 scrub starts 2024-03-04T16:16:07.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[29779]: 9.8 scrub ok 2024-03-04T16:16:07.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:16:07.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:16:07.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1257415569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:07.878 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:16:07.878 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[38113]: 9.8 scrub starts 2024-03-04T16:16:07.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[38113]: 9.8 scrub ok 2024-03-04T16:16:07.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:16:07.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:16:07.879 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:07 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1257415569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:07.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:07.917 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:16:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:07.945 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:07.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:07.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:16:07.994 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:16:08.037 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:16:08.041 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:16:08.041 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:16:08.041 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:16:08.041 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:16:08.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:16:08.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:16:08.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:16:08.042 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:16:08.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:08.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:07 smithi176 ceph-mon[33619]: 9.8 scrub starts 2024-03-04T16:16:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:07 smithi176 ceph-mon[33619]: 9.8 scrub ok 2024-03-04T16:16:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:07 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:16:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:07 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:16:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:07 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1257415569' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:08.205 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:08.238 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:08.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:08.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:08.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:08.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:08.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.648 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:08.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:08.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:08.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:08.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:08.917 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:08.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:16:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:16:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:16:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: pgmap v591: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 816 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:16:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: pgmap v591: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 816 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:08.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:09.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:09.054 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:16:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:16:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:16:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:16:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: pgmap v591: 148 pgs: 148 active+clean; 2.1 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 816 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:09.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:09.156 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:09.194 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:09.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.336 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:09.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:09.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:09.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:16:09.463 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:09.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.595 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:09.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:09.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.731 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:09.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:09.862 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:09.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:09.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.004 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:10.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:10.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:10.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:10.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:10.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:10.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 3.18 deep-scrub starts 2024-03-04T16:16:10.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 3.18 deep-scrub ok 2024-03-04T16:16:10.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:16:10.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 3.18 deep-scrub starts 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 3.18 deep-scrub ok 2024-03-04T16:16:10.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:16:10.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:16:10.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:16:10.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:16:10.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:16:10.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:09 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:16:10.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 3.18 deep-scrub starts 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 3.18 deep-scrub ok 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:16:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:09 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:16:10.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:10.430 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:10.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:10.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:10.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:10.687 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:16:10.687 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:16:10.707 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:10.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:10.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:10.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:10.983 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:10.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:10.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[29779]: pgmap v592: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[29779]: 6.19 deep-scrub starts 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[29779]: 2.1 deep-scrub starts 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[29779]: 2.1 deep-scrub ok 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[38113]: pgmap v592: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[38113]: 6.19 deep-scrub starts 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[38113]: 2.1 deep-scrub starts 2024-03-04T16:16:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:10 smithi038 ceph-mon[38113]: 2.1 deep-scrub ok 2024-03-04T16:16:11.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:11.119 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:11.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:11.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:10 smithi176 ceph-mon[33619]: pgmap v592: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:10 smithi176 ceph-mon[33619]: 6.19 deep-scrub starts 2024-03-04T16:16:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:10 smithi176 ceph-mon[33619]: 2.1 deep-scrub starts 2024-03-04T16:16:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:10 smithi176 ceph-mon[33619]: 2.1 deep-scrub ok 2024-03-04T16:16:11.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:11.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:11.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:11.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:11.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:11.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:16:11.650 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:11.685 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:11.685 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:11.787 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:11.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:11.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:11.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:11.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:12.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:12.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:12.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: 6.19 deep-scrub ok 2024-03-04T16:16:12.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:16:12.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:16:12.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: 6.19 deep-scrub ok 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:12.133 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:12.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:12.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:12.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: 6.19 deep-scrub ok 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:12.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:16:12.373 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:12.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:12.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:12.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:12.641 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:12.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:12.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:12.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:12.940 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:12.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: pgmap v593: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: osdmap e286: 8 total, 8 up, 6 in 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.9 deep-scrub starts 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[29779]: 6.9 deep-scrub ok 2024-03-04T16:16:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: pgmap v593: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: osdmap e286: 8 total, 8 up, 6 in 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.9 deep-scrub starts 2024-03-04T16:16:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:12 smithi038 ceph-mon[38113]: 6.9 deep-scrub ok 2024-03-04T16:16:13.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:13.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:13.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:13.324 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:13.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: pgmap v593: 148 pgs: 148 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:16:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:16:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:16:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: osdmap e286: 8 total, 8 up, 6 in 2024-03-04T16:16:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.9 deep-scrub starts 2024-03-04T16:16:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:12 smithi176 ceph-mon[33619]: 6.9 deep-scrub ok 2024-03-04T16:16:13.364 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:13.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:13.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:13.586 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:13.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:13.630 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:13.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:13.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.766 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:13.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:13.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:13.902 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:13.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:14.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:14.047 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:14.149 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: 9.9 deep-scrub starts 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: 9.9 deep-scrub starts 2024-03-04T16:16:14.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:16:14.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:13 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:16:14.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:14.317 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: 9.9 deep-scrub starts 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:16:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:13 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:16:14.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:14.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:14.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.593 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:14.687 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:14.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.724 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:14.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:14.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.861 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:14.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:14.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:14.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:14.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:14.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:15.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:15.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[29779]: pgmap v595: 148 pgs: 148 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[38113]: pgmap v595: 148 pgs: 148 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:16:15.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:14 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:16:15.273 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:14 smithi176 ceph-mon[33619]: pgmap v595: 148 pgs: 148 active+clean; 1.3 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-04T16:16:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:14 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:16:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:14 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:16:15.374 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:15.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:15.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:15.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:15.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:15.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:15.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.822 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:15.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:15.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:15.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:15.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:16:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:16:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:16:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:16:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:16:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:16:15.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:16:15.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:15 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:16:16.044 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:16.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:16.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:16.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:16.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:16.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:16.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:16:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:15 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:16:16.370 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:16.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:16.510 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:16.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:16.656 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:16.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:16.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:16.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:16.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:16.941 INFO:teuthology.orchestra.run.smithi038.stderr:clear 5.1e pg_upmap_items mapping 2024-03-04T16:16:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:16.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:17.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:17.073 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:17.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:17.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: pgmap v596: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]: dispatch 2024-03-04T16:16:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: pgmap v596: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]: dispatch 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:16:17.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:16 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:16:17.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:17.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: pgmap v596: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]: dispatch 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:16:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:16 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:16:17.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:17.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:17.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.620 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:17.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:17.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:17.753 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:17.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:17.900 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:17.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:18.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:18.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:18.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:18.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:18.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[29779]: 4.1c deep-scrub starts 2024-03-04T16:16:18.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[29779]: 4.1c deep-scrub ok 2024-03-04T16:16:18.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]': finished 2024-03-04T16:16:18.204 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[29779]: osdmap e287: 8 total, 8 up, 6 in 2024-03-04T16:16:18.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[38113]: 4.1c deep-scrub starts 2024-03-04T16:16:18.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[38113]: 4.1c deep-scrub ok 2024-03-04T16:16:18.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]': finished 2024-03-04T16:16:18.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:17 smithi038 ceph-mon[38113]: osdmap e287: 8 total, 8 up, 6 in 2024-03-04T16:16:18.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:18.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:18.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:17 smithi176 ceph-mon[33619]: 4.1c deep-scrub starts 2024-03-04T16:16:18.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:17 smithi176 ceph-mon[33619]: 4.1c deep-scrub ok 2024-03-04T16:16:18.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1348101507' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "5.1e"}]': finished 2024-03-04T16:16:18.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:17 smithi176 ceph-mon[33619]: osdmap e287: 8 total, 8 up, 6 in 2024-03-04T16:16:18.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:18.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:18.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:18.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:18.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.741 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:18.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:18.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:18.876 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:18.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:18.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.020 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:19.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:19.177 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: pgmap v598: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: osdmap e288: 8 total, 8 up, 6 in 2024-03-04T16:16:19.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:16:19.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:16:19.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:16:19.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:16:19.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:16:19.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:16:19.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:16:19.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:16:19.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:16:19.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:16:19.208 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: pgmap v598: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: osdmap e288: 8 total, 8 up, 6 in 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:16:19.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:18 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:16:19.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:19.313 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: pgmap v598: 148 pgs: 148 active+clean; 1.2 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 912 op/s 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: osdmap e288: 8 total, 8 up, 6 in 2024-03-04T16:16:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:16:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:16:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:16:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:16:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:16:19.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:18 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:16:19.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:19.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:19.559 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:19.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:19.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:19.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:19.878 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:19.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:19.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.011 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:20.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:20.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:20.145 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 9.9 deep-scrub ok 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:16:20.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 9.9 deep-scrub ok 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:16:20.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:19 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:16:20.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:20.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 9.9 deep-scrub ok 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:16:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:19 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:16:20.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:20.419 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:20.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:20.557 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:20.660 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:20.691 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:20.793 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:20.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:20.843 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:16:20.843 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:16:20.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:20.964 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:16:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:20.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:20.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:20.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: pgmap v600: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[29779]: 9.b scrub starts 2024-03-04T16:16:20.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: pgmap v600: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:16:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:20 smithi038 ceph-mon[38113]: 9.b scrub starts 2024-03-04T16:16:21.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:21.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:21.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:21.260 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:21.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: pgmap v600: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.77k op/s 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:16:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:16:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:16:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:16:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:20 smithi176 ceph-mon[33619]: 9.b scrub starts 2024-03-04T16:16:21.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:21.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:21.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:21.514 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:21.616 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:21.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:21.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:21.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:21.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:21.932 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:21.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:22.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.075 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:22.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:22.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:22.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:22.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:22.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: 9.b scrub ok 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:16:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:21 smithi176 ceph-mon[33619]: osdmap e289: 8 total, 8 up, 6 in 2024-03-04T16:16:22.433 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: 9.b scrub ok 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:21 smithi038 ceph-mon[29779]: osdmap e289: 8 total, 8 up, 6 in 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: 9.b scrub ok 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1855228870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:16:22.434 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:22 smithi038 ceph-mon[38113]: osdmap e289: 8 total, 8 up, 6 in 2024-03-04T16:16:22.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:22.516 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:22.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:22.653 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:22.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:22.791 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:22.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:22.925 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:22.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:23.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:23.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:23.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:23.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:23.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:16:23.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[29779]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:16:23.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:23.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:23.295 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[38113]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:16:23.295 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.295 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.295 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:23.295 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:23.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:23.340 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:23.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-mon[33619]: pgmap v602: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 815 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-04T16:16:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]: dispatch 2024-03-04T16:16:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:23.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:23.474 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:23.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:23.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:23.689 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:23.702 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:16:23.703 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:23.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:23.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:16:23.761 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:23.784 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:16:23.825 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:16:23.862 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:16:23.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:23.906 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:16:23.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:23.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:23.948 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:16:23.990 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:16:23.995 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:16:23.996 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:16:23.996 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:16:24.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:24.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:24.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:24.155 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:16:24.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 1 2024-03-04T16:16:24.188 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]': finished 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[29779]: osdmap e290: 8 total, 8 up, 6 in 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/274632327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]': finished 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[38113]: osdmap e290: 8 total, 8 up, 6 in 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/274632327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:24.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:24.349 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 3]}]': finished 2024-03-04T16:16:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:24 smithi176 ceph-mon[33619]: osdmap e290: 8 total, 8 up, 6 in 2024-03-04T16:16:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/274632327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:24.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:24.524 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:24.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:24.659 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:24.761 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:24.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:24.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:16:24.897 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:24.931 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:24.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:25.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:25.107 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:25.209 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:25.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:25.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:25.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:25 smithi176 ceph-mon[33619]: pgmap v604: 148 pgs: 148 active+clean; 548 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-03-04T16:16:25.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:25 smithi176 ceph-mon[33619]: osdmap e291: 8 total, 8 up, 6 in 2024-03-04T16:16:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:25 smithi038 ceph-mon[29779]: pgmap v604: 148 pgs: 148 active+clean; 548 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-03-04T16:16:25.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:25 smithi038 ceph-mon[29779]: osdmap e291: 8 total, 8 up, 6 in 2024-03-04T16:16:25.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:25 smithi038 ceph-mon[38113]: pgmap v604: 148 pgs: 148 active+clean; 548 MiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.76k op/s 2024-03-04T16:16:25.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:25 smithi038 ceph-mon[38113]: osdmap e291: 8 total, 8 up, 6 in 2024-03-04T16:16:25.394 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:25.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:25.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.533 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:25.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:25.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:25.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:25.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:25.808 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:25.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:25.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:25.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:26.048 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:26.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:26.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:26.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.370 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:26.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:26.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:26.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:26.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:26.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:26.782 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:26.884 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:26.904 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:26.968 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:26.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:27.070 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:27.095 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:27.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:27.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:27.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:27.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:27.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:27.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:27.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:27.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:27 smithi038 ceph-mon[29779]: pgmap v606: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:16:27.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:27 smithi038 ceph-mon[38113]: pgmap v606: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:16:27.509 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:27.604 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Removed 138488 objects 2024-03-04T16:16:27.604 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Clean up completed and total clean up time :106.032 2024-03-04T16:16:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:27 smithi176 ceph-mon[33619]: pgmap v606: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:16:27.611 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:27.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:27.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:27.759 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:27.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:27.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:27.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:27.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.013 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:28.114 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:28.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:28.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:28.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:28.269 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:28.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:28.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:28.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:28.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:28.658 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:28.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:28.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:28.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:16:28.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:28.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:28.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:28.968 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:29.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:29.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.062 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:29.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:29.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:29.188 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:29.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:29.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:29.439 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:29.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:29.565 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:29.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:29 smithi176 ceph-mon[33619]: pgmap v607: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:16:29.667 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.692 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:29.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:29 smithi038 ceph-mon[29779]: pgmap v607: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:16:29.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:29 smithi038 ceph-mon[38113]: pgmap v607: 148 pgs: 1 peering, 147 active+clean; 389 MiB data, 4.6 GiB used, 532 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:16:29.793 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:29.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:29.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:29.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:29.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:30.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:30.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:30.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:30.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:30.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:30.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:30.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:30.461 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:30.552 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:16:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:16:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:16:30.552 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:16:30.553 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:16:30.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:30.591 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:30.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:30.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:30.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.850 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:30.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:30.952 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:30.981 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:30.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:31.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:31.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:31.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:31.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:31.238 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:31.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:31.368 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:31.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:31.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:31.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:31 smithi038 ceph-mon[29779]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-04T16:16:31.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:31 smithi038 ceph-mon[38113]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-04T16:16:31.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:31.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:31.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:31 smithi176 ceph-mon[33619]: pgmap v608: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.63k op/s 2024-03-04T16:16:31.625 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:31.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:31.755 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:31.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:31.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:31.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:31.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:31.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:16:31.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:32.014 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:32.115 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:32.143 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:32.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:32.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.270 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:32.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:32.370 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:32.394 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:32.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:32.523 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:32.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:32.649 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:32.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:32.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:32.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:32.836 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-04T16:16:32.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-04T16:16:32.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:32.903 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:32.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:33.005 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:33.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:33.134 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:33.161 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:33.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:33.287 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:33.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:33 smithi176 ceph-mon[33619]: pgmap v609: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 597 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-04T16:16:33.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:33.389 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:33.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:33.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:33.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:33.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:33.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:33.671 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:33.697 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:33.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:33 smithi038 ceph-mon[29779]: pgmap v609: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 597 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-04T16:16:33.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:33 smithi038 ceph-mon[38113]: pgmap v609: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 597 B/s rd, 0 B/s wr, 1.18k op/s 2024-03-04T16:16:33.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:33.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:33.899 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:33.927 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:33.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:34.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:34.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:34.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:34.186 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:34.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:34.315 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:34.332 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T16:16:34.417 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:34.445 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:34.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:34.574 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:34.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:16:34.676 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:34.700 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:34.717 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:34.717 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:16:34.717 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:34.717 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:16:34.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:34.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:34.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:34.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:35.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:35.072 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:35.073 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:35.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:35.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:35.215 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:35.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:35.339 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:35.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:35.464 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:35.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.591 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:35.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: osdmap e292: 8 total, 8 up, 6 in 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/109146263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:35 smithi176 ceph-mon[33619]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:35.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:35.726 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:16:35.726 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: osdmap e292: 8 total, 8 up, 6 in 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/109146263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[29779]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: pgmap v610: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 876 op/s 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1864133858' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: osdmap e292: 8 total, 8 up, 6 in 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/109146263' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:35 smithi038 ceph-mon[38113]: from='client.35516 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:35.822 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:35.849 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:35.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 1 1 2024-03-04T16:16:35.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:35.979 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:35.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:36.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:36.108 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:36.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:36.233 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:36.307 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:16:36.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:36.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:36.465 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:36.493 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: pgmap v612: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 622 op/s 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[29779]: osdmap e293: 8 total, 8 up, 6 in 2024-03-04T16:16:36.590 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:36.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:36.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: pgmap v612: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 622 op/s 2024-03-04T16:16:36.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:36.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:36.591 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:36 smithi038 ceph-mon[38113]: osdmap e293: 8 total, 8 up, 6 in 2024-03-04T16:16:36.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: pgmap v612: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 622 op/s 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: from='client.35516 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:36 smithi176 ceph-mon[33619]: osdmap e293: 8 total, 8 up, 6 in 2024-03-04T16:16:36.622 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:36.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:36.748 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:36.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:36.874 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:36.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:36.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:37.004 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:37.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.130 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:37.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:37.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:37.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:37.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:37.389 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:37.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:37.515 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:37.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:37.645 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:37.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:37.775 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:37.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:37.904 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:37.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:38.006 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:38.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:38.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:38.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:38.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:38.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:38.304 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:16:38.327 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:16:38.347 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:16:38.366 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:16:38.387 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:16:38.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:38.407 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:16:38.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:38.426 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:16:38.447 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:16:38.449 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:16:38.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:38.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:38.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:38.674 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[29779]: pgmap v614: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1165345682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[38113]: pgmap v614: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1165345682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:38.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:38.803 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:38.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:38.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:38.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:39.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:39.063 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:38 smithi176 ceph-mon[33619]: pgmap v614: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:39.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1165345682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:39.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:39.192 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:39.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:39.319 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:39.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:39.444 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:39.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:39.570 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:39.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:39.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:39.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:39.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:39.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:16:39.926 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:39.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:39.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:39.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:40.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:40.076 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:40.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:40.201 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:40.302 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:40.309 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:40.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:40.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.456 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:40.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:40.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:40.581 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:40.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:40.698 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:40 smithi038 ceph-mon[29779]: pgmap v615: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:40.698 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:40 smithi038 ceph-mon[38113]: pgmap v615: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:40.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:40.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:40.837 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:40.939 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:40.967 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:40.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:41.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.094 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:41.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:41.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:40 smithi176 ceph-mon[33619]: pgmap v615: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:41.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:41.222 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:41.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:16:41.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:41.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:41.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:41.483 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:41.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:41.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:41.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:41.650 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_2' removed 2024-03-04T16:16:41.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:41.743 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:41.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:41.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:41.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:41.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:41.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:16:41.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:16:41.993 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:41.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:42.095 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:16:42.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:42.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:42.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.244 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:42.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:42.345 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:42.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:42.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:42.497 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:42.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:42.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:42.622 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:42.695 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:42.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:42.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[29779]: pgmap v616: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:16:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[29779]: osdmap e294: 8 total, 8 up, 6 in 2024-03-04T16:16:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[38113]: pgmap v616: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:16:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:42 smithi038 ceph-mon[38113]: osdmap e294: 8 total, 8 up, 6 in 2024-03-04T16:16:42.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.752 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:42.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:42.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:42.878 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:42.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:42.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.005 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:43.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:43.092 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:42 smithi176 ceph-mon[33619]: pgmap v616: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:16:43.092 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:42 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1700603624' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:16:43.092 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:42 smithi176 ceph-mon[33619]: osdmap e294: 8 total, 8 up, 6 in 2024-03-04T16:16:43.107 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:43.131 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:43.233 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.258 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:43.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:43.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:43.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:43.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:43.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:43.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:43.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:43.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:43.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:43.660 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:43.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:43.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:43.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:43.779 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-04T16:16:43.780 INFO:tasks.radosbench:Beginning radosbench... 2024-03-04T16:16:43.780 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-04T16:16:43.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-04T16:16:43.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:43.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:43.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:43.964 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:43.965 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:43.991 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:44.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:44.093 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:16:44.094 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:16:44.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:44.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.144 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:44.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:44.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:44.273 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:44.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:44.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:44.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:44.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:44.634 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:44.662 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:44.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:44 smithi038 ceph-mon[29779]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:44.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:44 smithi038 ceph-mon[38113]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:44.763 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:44.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:44.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:16:44.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:44.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:44.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:45.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:45.044 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:45.091 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:44 smithi176 ceph-mon[33619]: pgmap v618: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:45.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:45.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:45.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:45.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:45.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:45.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:45.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:45.549 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:45.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:45.675 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:45.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:45.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:45.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:45.927 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:45.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:46.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:46.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:46.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.179 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:46.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:46.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:46.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:46.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:46.429 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:46.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:46.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:46.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:46.591 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:46.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:46.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:46.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:46.805 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:46.867 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:46.908 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:46.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:46.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:47.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:47.059 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:47.149 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:16:47.149 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:16:47.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:47.187 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:47.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:47.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:47.303 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T16:16:47.311 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:16:47.316 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:47.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:47 smithi176 ceph-mon[33619]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 993 B/s rd, 0 op/s 2024-03-04T16:16:47.387 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:16:47.387 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1168598722' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.387 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:47 smithi176 ceph-mon[33619]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:47.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:47.543 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[29779]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 993 B/s rd, 0 op/s 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1168598722' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[29779]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[38113]: pgmap v619: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 993 B/s rd, 0 op/s 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1168598722' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:47 smithi038 ceph-mon[38113]: from='client.35540 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:16:47.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:47.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:47.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:47.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:47.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:47.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:47.823 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:16:47.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:47.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:47.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:48.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:48.076 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:48.110 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:48.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:48.201 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:48.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:48.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:48.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:48.456 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:48.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:48.558 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:16:48.571 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:48 smithi176 ceph-mon[33619]: osdmap e295: 8 total, 8 up, 6 in 2024-03-04T16:16:48.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:48.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:16:48.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:16:48.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:48.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:16:48.591 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[29779]: osdmap e295: 8 total, 8 up, 6 in 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1383181921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: from='client.35540 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:16:48.592 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:48 smithi038 ceph-mon[38113]: osdmap e295: 8 total, 8 up, 6 in 2024-03-04T16:16:48.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:48.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:48.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:48.833 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:48.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:48.959 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:48.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:49.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:49.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:49.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:49.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:49.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:49.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:49.423 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 3 1 2024-03-04T16:16:49.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:49.481 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:49 smithi176 ceph-mon[33619]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:16:49.490 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:49.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:49.615 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:49.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:49.718 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:16:49.718 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:16:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:49 smithi038 ceph-mon[29779]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:16:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:49 smithi038 ceph-mon[38113]: pgmap v621: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-04T16:16:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.744 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:49.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:49.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:49.872 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:49.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:49.998 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:49.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:50.100 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:50.124 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:50.226 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:50.250 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:50.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:50.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:16:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.376 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:50.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:50.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:50.501 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:50.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:16:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:50 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:16:50.615 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:50.626 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:16:50.636 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: pgmap v622: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:16:50.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:50 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:16:50.672 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:16:50.691 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:16:50.711 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:16:50.728 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:50.730 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:16:50.750 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:16:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.753 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:50.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:50.770 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:16:50.790 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:16:50.810 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:16:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:16:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:16:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:16:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:16:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:16:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:16:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:16:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:16:50.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:50.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:50.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:50.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:51.002 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:51.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:51.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:51.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:51.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:51.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:51.378 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:51.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:51.503 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:16:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:16:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/692207363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:51.587 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:16:51.587 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:16:51.587 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/692207363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:51.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:51 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:16:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:51 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:16:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/692207363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:51.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:51.671 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:51.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:51.756 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:51.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:16:51.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:51.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:51.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:51.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:52.007 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:52.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:52.134 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:52.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:52.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:52.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:52.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:52.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:52.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.511 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:52.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163755 7.4 7:20abbf33:::benchmark_data_smithi038_77863_object45487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:16:52.526 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 587.313146168, 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1434231819999998, 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163755 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.527 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211747+0000", 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211749+0000", 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4980000000000001e-06 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.528 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211755+0000", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9410000000000004e-06 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211767+0000", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1882000000000001e-05 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211775+0000", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9910000000000002e-06 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211836+0000", 2024-03-04T16:16:52.529 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1597e-05 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.211846+0000", 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6569999999999992e-06 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211462+0000", 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99961569800000005 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.530 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211477+0000", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4766999999999999e-05 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.337886+0000", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1264093390000001 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339470+0000", 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015841239999999999 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.531 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.339721+0000", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025078399999999999 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351914+0000", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012193641 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.351962+0000", 2024-03-04T16:16:52.532 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8062000000000002e-05 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352023+0000", 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0387999999999998e-05 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355003+0000", 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029802489999999999 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.533 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355055+0000", 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2178000000000001e-05 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355171+0000", 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000115386 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:163761 7.4 7:2ccd0a78:::benchmark_data_smithi038_77863_object45493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 587.29730850600004, 2024-03-04T16:16:52.534 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1332310689999998, 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 163761 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.535 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227585+0000", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227587+0000", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.812e-06 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227592+0000", 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.536 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227603+0000", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0251000000000001e-05 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227610+0000", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9650000000000002e-06 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227662+0000", 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2834999999999998e-05 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.537 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:05.227672+0000", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8989999999999996e-06 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211510+0000", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98383736899999996 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:06.211518+0000", 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5059999999999995e-06 2024-03-04T16:16:52.538 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352101+0000", 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.140582808 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352222+0000", 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120686 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.539 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,6", 2024-03-04T16:16:52.540 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.352361+0000", 2024-03-04T16:16:52.540 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000139345 2024-03-04T16:16:52.540 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.355375+0000", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0030134020000000001 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.356429+0000", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001054234 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.358238+0000", 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018089410000000001 2024-03-04T16:16:52.541 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360641+0000", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024032419999999999 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360703+0000", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2532000000000004e-05 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:07.360816+0000", 2024-03-04T16:16:52.542 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011275100000000001 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:177490 7.1 7:82ecc071:::benchmark_data_smithi038_77863_object59222:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 570.06108652399996, 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.10186015399999999, 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:16:52.543 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 177490 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463807+0000", 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.544 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463809+0000", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.767e-06 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463814+0000", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463824+0000", 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6409999999999997e-06 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.545 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.463831+0000", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1049999999999997e-06 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465019+0000", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011877140000000001 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465136+0000", 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011716400000000001 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.546 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.465249+0000", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011281399999999999 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.468002+0000", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027534370000000001 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.469006+0000", 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010038709999999999 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.547 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565552+0000", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.096545858999999998 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565616+0000", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4599999999999998e-05 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:07:22.565667+0000", 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0862999999999997e-05 2024-03-04T16:16:52.548 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14844.0:222378 7.1 7:83b38929:::benchmark_data_smithi038_77863_object104110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 518.42590125200002, 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.090076495000000006, 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14844", 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/1784052203", 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 222378 2024-03-04T16:16:52.549 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098992+0000", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.098995+0000", 2024-03-04T16:16:52.550 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6989999999999999e-06 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099001+0000", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9739999999999999e-06 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099026+0000", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4769999999999998e-05 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099034+0000", 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0760000000000003e-06 2024-03-04T16:16:52.551 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099091+0000", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6709000000000002e-05 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099742+0000", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00065097099999999997 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.099852+0000", 2024-03-04T16:16:52.552 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110408 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102653+0000", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028013769999999999 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.102881+0000", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000227686 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.188963+0000", 2024-03-04T16:16:52.553 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.086082045999999995 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189019+0000", 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6125000000000002e-05 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:08:14.189069+0000", 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9654000000000003e-05 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.554 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93725 8.2 8:4602f712:::benchmark_data_smithi038_107057_object93724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 406.29525467100001, 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12168869, 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93725 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.555 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229639+0000", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229641+0000", 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5799999999999999e-06 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.556 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229692+0000", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1480999999999997e-05 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229701+0000", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4099999999999997e-06 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229714+0000", 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2140999999999999e-05 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.557 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229749+0000", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5058000000000002e-05 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229848+0000", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9230999999999999e-05 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.229968+0000", 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000119874 2024-03-04T16:16:52.558 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.234094+0000", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041263849999999998 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237005+0000", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029105889999999999 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351181+0000", 2024-03-04T16:16:52.559 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114175783 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351277+0000", 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6447999999999995e-05 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351328+0000", 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0710000000000001e-05 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.560 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93728 8.2 8:447b8219:::benchmark_data_smithi038_107057_object93727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 406.29424869500002, 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.12084650500000001, 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93728 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.561 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230645+0000", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230646+0000", 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1030000000000001e-06 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.562 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230704+0000", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8165999999999998e-05 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230714+0000", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9469999999999998e-06 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.230721+0000", 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1579999999999999e-06 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.563 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.231787+0000", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010659899999999999 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.231921+0000", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013319300000000001 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.232063+0000", 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014256199999999999 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.564 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.234204+0000", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002141034 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.239004+0000", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004799902 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351370+0000", 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.11236549999999999 2024-03-04T16:16:52.565 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351455+0000", 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5463999999999994e-05 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351491+0000", 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6486000000000003e-05 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.566 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:93733 8.2 8:45cfc703:::benchmark_data_smithi038_107057_object93732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 406.28791118499998, 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.114652037, 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 93733 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:16:52.567 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236982+0000", 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.236984+0000", 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5939999999999999e-06 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.568 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237039+0000", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5034999999999997e-05 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237046+0000", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0469999999999996e-06 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237051+0000", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4569999999999998e-06 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237077+0000", 2024-03-04T16:16:52.569 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6139000000000002e-05 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237169+0000", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2542999999999998e-05 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.237262+0000", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3041999999999995e-05 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.239083+0000", 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001821079 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.570 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.241678+0000", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002594963 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351528+0000", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.109849579 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351601+0000", 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3014000000000003e-05 2024-03-04T16:16:52.571 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:06.351634+0000", 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3544999999999998e-05 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:114775 8.5 8:aa81b258:::benchmark_data_smithi038_107057_object114774:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 384.27769112700003, 2024-03-04T16:16:52.572 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1283407419999998, 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 114775 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.573 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247203+0000", 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247205+0000", 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.249e-06 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247248+0000", 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3449999999999999e-05 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.574 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247261+0000", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2761e-05 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247271+0000", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6320000000000003e-06 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247347+0000", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6029e-05 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.575 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.247358+0000", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1127999999999999e-05 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.232977+0000", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98561953400000002 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.232991+0000", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3873000000000001e-05 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.576 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.350151+0000", 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1171602970000001 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.351593+0000", 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014410849999999999 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,6,7", 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.351847+0000", 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025436700000000003 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.577 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.364523+0000", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012676031000000001 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.364591+0000", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8526000000000001e-05 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.367558+0000", 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029666889999999998 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.578 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375350+0000", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007791765 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375422+0000", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1966999999999997e-05 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:30.375543+0000", 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121359 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.579 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1839 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 93955426377600 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 383.66627440899998, 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.47448410200000002, 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1839 2024-03-04T16:16:52.580 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858619+0000", 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.581 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858620+0000", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.221e-06 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858625+0000", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5979999999999999e-06 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858634+0000", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0040000000000005e-06 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.582 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858711+0000", 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6966999999999997e-05 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858832+0000", 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121279 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858842+0000", 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8330000000000006e-06 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.583 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.322095+0000", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.46325266599999998 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.322485+0000", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039060800000000003 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.326605+0000", 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041195490000000001 2024-03-04T16:16:52.584 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.326845+0000", 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00024025300000000001 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.333103+0000", 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062581240000000003 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.585 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.14478.0:1841 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93955430886784 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 383.66609664100002, 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.476828576, 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.14478", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/2081174426", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 1841 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.586 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858797+0000", 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858798+0000", 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7000000000000003e-07 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.587 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858802+0000", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8349999999999997e-06 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858809+0000", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7200000000000006e-06 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858815+0000", 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5500000000000002e-06 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.588 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858864+0000", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8927000000000001e-05 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:28.858871+0000", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3069999999999997e-06 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.329016+0000", 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.470145074 2024-03-04T16:16:52.589 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.329355+0000", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000338573 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.331135+0000", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017796419999999999 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.331607+0000", 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000472325 2024-03-04T16:16:52.590 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:29.335626+0000", 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040187529999999999 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:115508 8.5 8:a9a17dc4:::benchmark_data_smithi038_107057_object115507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 381.38512873000002, 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080985018000000006, 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.591 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.592 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.592 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 115508 2024-03-04T16:16:52.592 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139765+0000", 2024-03-04T16:16:52.593 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139767+0000", 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7940000000000001e-06 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139795+0000", 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8564999999999998e-05 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.594 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139803+0000", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7330000000000003e-06 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139810+0000", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5060000000000001e-06 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139904+0000", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4498000000000002e-05 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.595 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140045+0000", 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014108400000000001 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,6,7", 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.141168+0000", 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001123209 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147773+0000", 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006605026 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.596 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147801+0000", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7906e-05 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.147809+0000", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2300000000000008e-06 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220629+0000", 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072819350000000005 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.597 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220666+0000", 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7020999999999997e-05 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220750+0000", 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4096000000000003e-05 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.598 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.15171.0:115509 8.2 8:447c5fda:::benchmark_data_smithi038_107057_object115508:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 381.38507045199998, 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.080756245000000004, 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.15171", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 115509 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.599 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139823+0000", 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139824+0000", 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8599999999999997e-07 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.600 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139885+0000", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0838999999999998e-05 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139891+0000", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0900000000000001e-06 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139897+0000", 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3960000000000003e-06 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.601 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.139960+0000", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2432000000000001e-05 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140080+0000", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012054500000000001 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.140227+0000", 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014703200000000001 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.602 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.145807+0000", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.005580018 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.148427+0000", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002619466 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220459+0000", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072032465000000004 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.603 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220525+0000", 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5420999999999996e-05 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:10:31.220579+0000", 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4654999999999998e-05 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:167295 9.c e278/242 9:3260cdd2:::benchmark_data_smithi038_131582_object28804:head v 278'10651, mlcod=278'10648)", 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:03.586875+0000", 2024-03-04T16:16:52.604 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 108.938019059, 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028729007000000001, 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586875+0000", 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586875+0000", 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.605 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586876+0000", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4079999999999999e-06 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586884+0000", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9209999999999992e-06 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586895+0000", 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1399e-05 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.606 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586903+0000", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6620000000000005e-06 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586957+0000", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3528e-05 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.586977+0000", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.012e-05 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.607 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.615562+0000", 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028585147000000002 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:03.615604+0000", 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1822e-05 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:168095 9.4 e278/242 9:2bb909fa:::benchmark_data_smithi038_131582_object29604:head v 278'11462, mlcod=278'11454)", 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:04.298179+0000", 2024-03-04T16:16:52.608 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 108.22671416599999, 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030592122999999999, 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298179+0000", 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298179+0000", 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.609 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298181+0000", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2669999999999999e-06 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298188+0000", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8299999999999998e-06 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298201+0000", 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3424999999999999e-05 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.610 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298209+0000", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8350000000000004e-06 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298263+0000", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4471000000000001e-05 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.298293+0000", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9498000000000001e-05 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.611 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.328739+0000", 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.030446309000000001 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:04.328772+0000", 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2487999999999998e-05 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:180400 9.4 e279/242 9:29151d87:::benchmark_data_smithi038_131582_object41909:head v 279'14928, mlcod=279'14920)", 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:16.390947+0000", 2024-03-04T16:16:52.612 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 96.133946772000002, 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027333803, 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390947+0000", 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390947+0000", 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.613 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390948+0000", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.116e-06 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390959+0000", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1488000000000001e-05 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390969+0000", 2024-03-04T16:16:52.614 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8700000000000004e-06 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.390974+0000", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9259999999999999e-06 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.406725+0000", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015751134 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.406746+0000", 2024-03-04T16:16:52.615 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1061e-05 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.418238+0000", 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011491448 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:16.418281+0000", 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2759999999999997e-05 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.616 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:182102 9.4 e279/242 9:2c77bdb3:::benchmark_data_smithi038_131582_object43611:head v 279'15476, mlcod=279'15472)", 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:18.130832+0000", 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 94.394061180999998, 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.028470509000000001, 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130832+0000", 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.617 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130832+0000", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130834+0000", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1650000000000001e-06 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130841+0000", 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9750000000000001e-06 2024-03-04T16:16:52.618 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130855+0000", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4209000000000001e-05 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.130862+0000", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.99e-06 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.131629+0000", 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076706500000000004 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.619 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.131654+0000", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4782000000000001e-05 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.159212+0000", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.027558450000000002 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:18.159303+0000", 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0872999999999995e-05 2024-03-04T16:16:52.620 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:221363 9.e e282/242 9:7c6d25db:::benchmark_data_smithi038_131582_object82872:head v 282'13735, mlcod=282'13733)", 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:47.526371+0000", 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.998522288000004, 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016627336, 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526371+0000", 2024-03-04T16:16:52.621 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526371+0000", 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526373+0000", 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.536e-06 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.622 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526385+0000", 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1847e-05 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526399+0000", 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3906999999999999e-05 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526406+0000", 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3130000000000001e-06 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.623 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526483+0000", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7250999999999996e-05 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.526506+0000", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3271000000000001e-05 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.542804+0000", 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01629717 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.624 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.542999+0000", 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019504099999999999 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "rep_scrubmap(6.e e190 from shard 5)", 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:47.527592+0000", 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.997301393000001, 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.015410567, 2024-03-04T16:16:52.625 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527592+0000", 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527592+0000", 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.626 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527593+0000", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5199999999999995e-07 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527598+0000", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5360000000000003e-06 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527609+0000", 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1141000000000001e-05 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.627 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.527614+0000", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3900000000000001e-06 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.542521+0000", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014906519999999999 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.542555+0000", 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4140000000000002e-05 2024-03-04T16:16:52.628 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.543003+0000", 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00044798800000000002 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:221374 9.e e282/242 9:7e52128e:::benchmark_data_smithi038_131582_object82883:head v 282'13736, mlcod=282'13733)", 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:47.532334+0000", 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.992560107000003, 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014286368000000001, 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.629 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532334+0000", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532334+0000", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532335+0000", 2024-03-04T16:16:52.630 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4160000000000001e-06 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532343+0000", 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8849999999999999e-06 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532357+0000", 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4270999999999999e-05 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.631 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.532364+0000", 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3900000000000004e-06 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.540334+0000", 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079693560000000004 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.540353+0000", 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9040999999999999e-05 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.632 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.546477+0000", 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0061243290000000004 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:47.546620+0000", 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014268000000000001 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.15456.0:232270 9.1 e282/242 9:8ab4b816:::benchmark_data_smithi038_131582_object93779:head v 282'14500, mlcod=282'14494)", 2024-03-04T16:16:52.633 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:15:55.256322+0000", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.268571367, 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013599128, 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256322+0000", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256322+0000", 2024-03-04T16:16:52.634 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256324+0000", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2619999999999999e-06 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256331+0000", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6299999999999998e-06 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256340+0000", 2024-03-04T16:16:52.635 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9190000000000005e-06 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.256347+0000", 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8079999999999999e-06 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.260961+0000", 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046140310000000002 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.636 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.260980+0000", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9479000000000001e-05 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.269843+0000", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088623810000000008 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:15:55.269921+0000", 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8617999999999998e-05 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.637 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:16:52.638 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:16:52.638 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:16:52.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_3' created 2024-03-04T16:16:52.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:52.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:52.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:52 smithi176 ceph-mon[33619]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:52 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:16:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:52 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:16:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:52 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-04T16:16:52.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:52.918 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:52.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:52.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:52.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[29779]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:52.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[38113]: pgmap v623: 132 pgs: 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:16:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:52 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-04T16:16:53.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:53.042 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:53.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:53.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:53.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:53.290 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:53.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:16:53.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:16:53.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:53.451 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:16:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:16:53.451 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:53.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-04T16:16:53.489 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:53.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:53.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: osdmap e296: 8 total, 8 up, 6 in 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:53.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:16:53.626 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:53.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:53.702 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: osdmap e296: 8 total, 8 up, 6 in 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:53.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2263048457' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: osdmap e296: 8 total, 8 up, 6 in 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:16:53.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:16:53.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:53.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:53.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:53.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:53.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:54.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:54.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:54.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:54.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:54.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:54.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:54.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:54.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:54.580 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:54.682 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:54.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: pgmap v625: 148 pgs: 11 creating+peering, 5 unknown, 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: osdmap e297: 8 total, 8 up, 6 in 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: pgmap v625: 148 pgs: 11 creating+peering, 5 unknown, 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: osdmap e297: 8 total, 8 up, 6 in 2024-03-04T16:16:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:16:54.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:16:54.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:16:54.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:54 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:16:54.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:54.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:54.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:54.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: pgmap v625: 148 pgs: 11 creating+peering, 5 unknown, 132 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:16:54.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:16:54.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: osdmap e297: 8 total, 8 up, 6 in 2024-03-04T16:16:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:16:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:16:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:16:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:54 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:16:54.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:54.955 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:54.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:55.056 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:55.181 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:55.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:55.209 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:16:55.209 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:16:55.210 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-03-04T16:16:55.210 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:16:55.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:55.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:55.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:55.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:55.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:55.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:55.584 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:55.594 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:16:55.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:55.692 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T16:16:55.706 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:55.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:16:55.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:16:55.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[29779]: osdmap e298: 8 total, 8 up, 6 in 2024-03-04T16:16:55.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3230716463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:55.807 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[29779]: from='client.35558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:55.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:16:55.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:16:55.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[38113]: osdmap e298: 8 total, 8 up, 6 in 2024-03-04T16:16:55.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3230716463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:55.808 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:55 smithi038 ceph-mon[38113]: from='client.35558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:55.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:55.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:55.881 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:16:55.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:55.959 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:55.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:55.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:56.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:56.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:55 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:16:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:55 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:16:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:55 smithi176 ceph-mon[33619]: osdmap e298: 8 total, 8 up, 6 in 2024-03-04T16:16:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3230716463' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:55 smithi176 ceph-mon[33619]: from='client.35558 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:16:56.128 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:56.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:56.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:56.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:56.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:16:56.337 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:16:56.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:56.467 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:56.569 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.593 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:56.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:56.694 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:56.720 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:56.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:56.846 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: pgmap v628: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: from='client.35558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: osdmap e299: 8 total, 8 up, 6 in 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:16:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:56 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:16:56.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:56.971 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:56.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: pgmap v628: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: from='client.35558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: osdmap e299: 8 total, 8 up, 6 in 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:16:56.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: pgmap v628: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: from='client.35558 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: osdmap e299: 8 total, 8 up, 6 in 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:16:56.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:56 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:16:57.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:57.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:57.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:57.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:57.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:57.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:57.362 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 4 1 2024-03-04T16:16:57.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:57.473 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:57.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:57.598 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:57.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:57.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:57.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:57.818 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:16:57.818 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:16:57.818 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:16:57.818 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:16:57.818 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:16:57.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:16:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:57 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:16:57.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:16:57.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:16:57.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:16:57.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:16:57.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:16:57.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:57 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:16:57.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:57.974 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:57.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:58.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:58.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:16:58.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:16:58.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:16:58.089 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:16:58.092 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:16:58.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:58.113 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:16:58.133 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:16:58.153 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:16:58.154 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:16:58.154 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:16:58.154 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:16:58.154 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:16:58.155 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:16:58.155 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:16:58.155 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:16:58.155 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:16:58.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:58.226 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:58.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:58.350 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:58.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:58.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:16:58.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:58.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:58.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:58.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:58.678 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:16:58.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:58.729 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: pgmap v630: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3318726762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2613373095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:58.790 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:58 smithi176 ceph-mon[33619]: from='client.25534 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:58.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:16:58.854 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:16:58.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:16:58.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:58.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:58.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:16:58.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: pgmap v630: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3318726762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2613373095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[29779]: from='client.25534 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: pgmap v630: 148 pgs: 14 creating+peering, 134 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-04T16:16:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3318726762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:16:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2613373095' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:58 smithi038 ceph-mon[38113]: from='client.25534 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:16:59.039 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:16:59.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:59.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:59.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:16:59.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:16:59.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:59.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:59.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:16:59.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:16:59.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:16:59.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:16:59.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:16:59.720 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:59 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:59.720 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:59 smithi176 ceph-mon[33619]: from='client.25534 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:59.720 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:16:59 smithi176 ceph-mon[33619]: osdmap e300: 8 total, 8 up, 6 in 2024-03-04T16:16:59.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:16:59.833 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:59.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:16:59.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:16:59.958 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:16:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:16:59.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:16:59.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:59.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[29779]: from='client.25534 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:59.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[29779]: osdmap e300: 8 total, 8 up, 6 in 2024-03-04T16:16:59.963 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:16:59.963 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[38113]: from='client.25534 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:16:59.963 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:16:59 smithi038 ceph-mon[38113]: osdmap e300: 8 total, 8 up, 6 in 2024-03-04T16:17:00.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:00.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:00.185 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:00.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:00.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:00.314 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:17:00.314 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:17:00.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:00.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:00.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:00.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:00.589 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:00.691 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:00.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:00.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:00.839 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:00.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:00.967 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:00.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: pgmap v632: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 860 B/s rd, 0 op/s 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:17:00.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:00 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: pgmap v632: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 860 B/s rd, 0 op/s 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:17:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: pgmap v632: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 860 B/s rd, 0 op/s 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:17:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:00 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:17:00.998 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:01.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:01.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:01.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:17:01.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:01.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:01.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:01.346 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:01.351 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:01.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:01.472 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:01.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:01.586 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:17:01.597 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:01.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:01.711 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-04T16:17:01.724 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:01.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:01.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:01.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:17:01.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:17:01.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:17:01.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:17:01.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3259246318' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[29779]: from='client.35576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3259246318' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:01.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:01 smithi038 ceph-mon[38113]: from='client.35576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:01.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:01.977 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:01.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:02.078 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:02.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3259246318' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:01 smithi176 ceph-mon[33619]: from='client.35576 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:17:02.204 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:02.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.229 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:02.330 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:02.353 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:02.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:02.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:02.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:02.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:02.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:02.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:02.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:17:02.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:02.857 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:02.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:02.982 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:02.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:02.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:02.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 737 B/s rd, 0 op/s 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: from='client.35576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: osdmap e301: 8 total, 8 up, 6 in 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 737 B/s rd, 0 op/s 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: from='client.35576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: osdmap e301: 8 total, 8 up, 6 in 2024-03-04T16:17:02.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:17:02.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:02 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:17:03.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: pgmap v633: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 737 B/s rd, 0 op/s 2024-03-04T16:17:03.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:17:03.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:17:03.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: from='client.35576 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:17:03.054 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: osdmap e301: 8 total, 8 up, 6 in 2024-03-04T16:17:03.054 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:17:03.054 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:02 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:17:03.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:03.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:03.210 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:03.219 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:03.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:03.308 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:17:03.309 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:17:03.309 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:03.335 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:03.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:03.429 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:03.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:03.485 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:03.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:03.610 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:03.659 DEBUG:teuthology.orchestra.run.smithi038:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-04T16:17:03.661 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-04T16:17:03.661 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-04T16:17:03.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.737 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:03.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:03.840 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:03.864 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:03.966 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:03.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:03 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:17:03.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:03 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:17:03.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:03.990 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:03.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:03.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:17:03.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:17:03.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:17:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:17:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:03.998 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:hints = 1 2024-03-04T16:17:03.998 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-04T16:17:03.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object prefix: benchmark_data_smithi038_155391 2024-03-04T16:17:03.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:17:03.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 0 0 0 0 0 0 - 0 2024-03-04T16:17:04.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:17:04.092 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:04.133 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:04.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:04.273 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:04.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:04.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.421 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:04.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:04.482 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:04.504 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:04.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:04.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:04.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:04.698 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:04.800 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:04.835 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:04.900 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:17:04.900 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:17:04.901 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:04 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:17:04.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:04.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:04.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:17:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: pgmap v635: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:17:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:04 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:17:04.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 1 16 3577 3561 222.507 222.562 0.00260685 0.00447097 2024-03-04T16:17:05.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:05.120 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:05.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:05.259 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:05.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:05.401 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:05.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:05.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:05.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:05.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-04T16:17:05.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:05.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:05.917 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:05.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3167936601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:05 smithi176 ceph-mon[33619]: from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:05 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:17:05.931 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:05 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:17:05.953 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:05.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3167936601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[29779]: from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:17:05.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:17:05.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3167936601' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[38113]: from='client.25555 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:17:05.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:17:05.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:05 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:17:05.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 2 16 6764 6748 210.839 199.188 0.00432046 0.00472896 2024-03-04T16:17:06.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:06.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.094 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:06.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:06.100 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:17:06.124 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:17:06.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:17:06.187 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:17:06.196 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:06.229 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:17:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.234 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:06.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:06.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:17:06.307 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:17:06.335 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:06.347 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:17:06.351 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:17:06.352 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:17:06.352 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:17:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.373 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:06.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:06.433 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:17:06.433 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:17:06.433 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:17:06.433 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:17:06.434 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:17:06.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:06.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:06.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:06.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:06.756 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:06.790 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:06.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:06.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:06.932 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:06.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:06.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: 4.f deep-scrub starts 2024-03-04T16:17:06.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: 4.f deep-scrub ok 2024-03-04T16:17:06.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: osdmap e302: 8 total, 8 up, 6 in 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1865522355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: 4.f deep-scrub starts 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: 4.f deep-scrub ok 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: osdmap e302: 8 total, 8 up, 6 in 2024-03-04T16:17:06.983 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1865522355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:06.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 3 16 9662 9646 200.93 181.125 0.00638865 0.00496836 2024-03-04T16:17:07.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:07.071 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:07.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: 4.f deep-scrub starts 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: 4.f deep-scrub ok 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: pgmap v636: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: from='client.25555 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: osdmap e302: 8 total, 8 up, 6 in 2024-03-04T16:17:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1865522355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:07.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:17:07.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:07.209 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:07.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:07.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:07.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:07.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:07.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:07.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:07.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 5 1 2024-03-04T16:17:07.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:07.619 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:07.635 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:07.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:07.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:07.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:07.920 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:07.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:07.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 4 16 12636 12620 197.161 185.875 0.00317914 0.00506572 2024-03-04T16:17:08.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:17:08.066 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:07 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:17:08.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:08.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:08.210 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:08.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:08.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:17:08.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:17:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:07 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:17:08.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:08.350 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:08.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:08.504 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:08.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.645 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:08.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:08.746 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:08.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: pgmap v638: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:08.848 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: 5.1f deep-scrub starts 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[29779]: 5.1f deep-scrub ok 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: pgmap v638: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:08.849 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: 5.1f deep-scrub starts 2024-03-04T16:17:08.850 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:08 smithi038 ceph-mon[38113]: 5.1f deep-scrub ok 2024-03-04T16:17:08.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:08.925 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:08.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:09.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 5 16 15449 15433 192.888 175.812 0.00925882 0.00517783 2024-03-04T16:17:09.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:09.065 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:09.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: pgmap v638: 148 pgs: 148 active+clean; 582 KiB data, 3.3 GiB used, 533 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: 5.1f deep-scrub starts 2024-03-04T16:17:09.106 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:08 smithi176 ceph-mon[33619]: 5.1f deep-scrub ok 2024-03-04T16:17:09.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:09.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:09.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:09.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:09.327 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:09.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:09.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:09.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:09.472 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:09.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:09.612 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:09.669 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:17:09.670 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:17:09.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:17:09.670 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [] 2024-03-04T16:17:09.670 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:17:09.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:09.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:09.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:09.811 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:17:09.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:09.891 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:09.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:09.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[29779]: osdmap e303: 8 total, 8 up, 6 in 2024-03-04T16:17:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/649542614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[29779]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[38113]: osdmap e303: 8 total, 8 up, 6 in 2024-03-04T16:17:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/649542614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:09 smithi038 ceph-mon[38113]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:09.999 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 6 15 18011 17996 187.436 160.188 0.0116737 0.00532762 2024-03-04T16:17:10.025 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:09 smithi176 ceph-mon[33619]: osdmap e303: 8 total, 8 up, 6 in 2024-03-04T16:17:10.025 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/649542614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:10.025 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:09 smithi176 ceph-mon[33619]: from='client.25564 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:17:10.030 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:10.131 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:10.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:10.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:10.309 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:10.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:17:10.410 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:10.549 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:10.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:10.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:10.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:10.803 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:10.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:10.879 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:10.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:10.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:10.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:10.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: pgmap v640: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[29779]: osdmap e304: 8 total, 8 up, 6 in 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: pgmap v640: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:17:10.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:10 smithi038 ceph-mon[38113]: osdmap e304: 8 total, 8 up, 6 in 2024-03-04T16:17:11.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 7 16 20538 20522 183.204 157.875 0.00427727 0.00545206 2024-03-04T16:17:11.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:11.032 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:17:11.033 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:17:11.033 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: pgmap v640: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-04T16:17:11.033 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:11.033 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: from='client.25564 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:17:11.033 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:10 smithi176 ceph-mon[33619]: osdmap e304: 8 total, 8 up, 6 in 2024-03-04T16:17:11.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:11.163 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:11.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:11.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:17:11.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:11.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:11.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:11.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:11.695 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:11.733 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:11.810 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:11.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:11.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:11.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:11.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:12.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 8 15 23018 23003 179.685 155.062 0.00602346 0.00555964 2024-03-04T16:17:12.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:12.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:12.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:12.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:12.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:12.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:12.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:12.449 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:12.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:12.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.576 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:12.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:12.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:12.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:12.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:12.843 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:12.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:12.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:12.982 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:12.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:13.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 9 16 24948 24932 173.113 120.562 0.00191657 0.00576263 2024-03-04T16:17:13.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:12 smithi176 ceph-mon[33619]: pgmap v642: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:17:13.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:13.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:17:13.211 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:12 smithi038 ceph-mon[29779]: pgmap v642: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:17:13.211 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:12 smithi038 ceph-mon[38113]: pgmap v642: 148 pgs: 148 active+clean; 824 MiB data, 5.6 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-04T16:17:13.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:13.261 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:13.337 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:17:13.337 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:17:13.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:25592 10.c 10:34afea2b:::benchmark_data_smithi038_155391_object25591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:17:13.330573+0000", 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0042050830000000001, 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0042312490000000003, 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:17:13.338 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25592 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330573+0000", 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:17:13.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330573+0000", 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330574+0000", 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.02e-07 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330606+0000", 2024-03-04T16:17:13.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2156999999999997e-05 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330614+0000", 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8800000000000008e-06 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330621+0000", 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.596e-06 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330698+0000", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.674e-05 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330799+0000", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000101376 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,4", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.330930+0000", 2024-03-04T16:17:13.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131056 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.333512+0000", 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025819580000000001 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.333551+0000", 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9147999999999999e-05 2024-03-04T16:17:13.343 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:25596 10.c 10:3976b605:::benchmark_data_smithi038_155391_object25595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:17:13.331013+0000", 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0037649739999999999, 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038491469999999998, 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:17:13.344 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 25596 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331013+0000", 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:17:13.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331013+0000", 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331020+0000", 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.144e-06 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331070+0000", 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0676000000000001e-05 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331082+0000", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.221e-05 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.331090+0000", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1729999999999998e-06 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.332145+0000", 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001055369 2024-03-04T16:17:13.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.332267+0000", 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000122163 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,4", 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.332437+0000", 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017020300000000001 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:17:13.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:17:13.333523+0000", 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0010854250000000001 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 2 2024-03-04T16:17:13.349 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:17:13.364 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:13.405 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:13.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:13.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:13.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:13.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:13.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:13.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:13.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:13.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.811 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:13.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:13.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:13.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:13.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:14.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 10 16 26921 26905 168.132 123.312 0.00617907 0.00594185 2024-03-04T16:17:14.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:14.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:14.079 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.087 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:14.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:14.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:14.209 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:17:14.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:14.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.377 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:14.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:14.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:14.524 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:14.618 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:14.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:14.661 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:14.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:14.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.801 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:14.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:14.826 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-04T16:17:14.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:14.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:14.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:14.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[29779]: pgmap v643: 148 pgs: 148 active+clean; 1.3 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-04T16:17:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3101046084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[29779]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[38113]: pgmap v643: 148 pgs: 148 active+clean; 1.3 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-04T16:17:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3101046084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:14 smithi038 ceph-mon[38113]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:15.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 11 15 29095 29080 165.204 135.938 0.00788788 0.00604769 2024-03-04T16:17:15.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:14 smithi176 ceph-mon[33619]: pgmap v643: 148 pgs: 148 active+clean; 1.3 GiB data, 7.3 GiB used, 529 GiB / 536 GiB avail; 1.3 KiB/s rd, 173 MiB/s wr, 2.77k op/s 2024-03-04T16:17:15.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3101046084' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:15.039 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:14 smithi176 ceph-mon[33619]: from='client.35606 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:17:15.041 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:15.080 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:15.182 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:15.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:15.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:15.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:15.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:15.501 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:15.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:15.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:15.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:15.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:15.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:15.939 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:15.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:15.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:15 smithi038 ceph-mon[38113]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:17:15.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:15 smithi038 ceph-mon[38113]: osdmap e305: 8 total, 8 up, 6 in 2024-03-04T16:17:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:15 smithi038 ceph-mon[29779]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:17:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:15 smithi038 ceph-mon[29779]: osdmap e305: 8 total, 8 up, 6 in 2024-03-04T16:17:16.000 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 12 16 31213 31197 162.462 132.312 0.00484248 0.00614304 2024-03-04T16:17:16.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:16.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:15 smithi176 ceph-mon[33619]: from='client.35606 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:17:16.053 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:15 smithi176 ceph-mon[33619]: osdmap e305: 8 total, 8 up, 6 in 2024-03-04T16:17:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:16.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.212 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:16.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:16.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:16.339 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:16.440 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:16.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:16.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:16.612 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:16.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:16.741 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:16.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:16.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:16.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:16.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:17.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 13 16 32713 32697 157.176 93.75 0.00746308 0.0063327 2024-03-04T16:17:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.007 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:17.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:17.107 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:16 smithi176 ceph-mon[33619]: pgmap v645: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 749 B/s rd, 128 MiB/s wr, 2.06k op/s 2024-03-04T16:17:17.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:17.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:17.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:16 smithi038 ceph-mon[29779]: pgmap v645: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 749 B/s rd, 128 MiB/s wr, 2.06k op/s 2024-03-04T16:17:17.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:16 smithi038 ceph-mon[38113]: pgmap v645: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 749 B/s rd, 128 MiB/s wr, 2.06k op/s 2024-03-04T16:17:17.270 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:17.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:17.410 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:17.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 7 1 2024-03-04T16:17:17.491 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:17.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.563 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:17.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:17.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:17.695 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:17.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:17.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:17.882 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:17:17.882 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:17:17.882 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:17:17.882 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:17:17.882 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:17:17.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:17.963 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:17.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:18.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 14 16 34243 34227 152.779 95.625 0.00344372 0.00652029 2024-03-04T16:17:18.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:18.089 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:18.149 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:17:18.178 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:18.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:18.197 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:17:18.237 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:17:18.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:18.262 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:17:18.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:18.346 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:17:18.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:18.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:18.384 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:17:18.424 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:17:18.465 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:17:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:17:18.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:18.515 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:18.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:18.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:18.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.642 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:18.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:18.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.791 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:18.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:18.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:18.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:18.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:19.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 15 16 35598 35582 148.239 84.6875 0.00618524 0.00674059 2024-03-04T16:17:19.029 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:19.070 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:19.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:19.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:19.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:18 smithi038 ceph-mon[29779]: pgmap v646: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:19.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1235123330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:19.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:18 smithi038 ceph-mon[38113]: pgmap v646: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:19.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1235123330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:19.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.335 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:18 smithi176 ceph-mon[33619]: pgmap v646: 148 pgs: 148 active+clean; 1.5 GiB data, 7.9 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1235123330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:19.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:19.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:19.590 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:19.692 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.730 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:19.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:19.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:19.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:19.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:19.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:20.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 16 16 37060 37044 144.685 91.375 0.00985861 0.00690641 2024-03-04T16:17:20.006 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:20.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:20.143 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:20.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:20.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:20.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:20.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:20.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:20.558 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:20.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:20.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:20.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:20.828 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:20.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:20.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:20.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:21.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 17 16 38589 38573 141.795 95.5625 0.0058245 0.00704724 2024-03-04T16:17:21.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:21.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:21.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:21.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:21.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:21.350 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:21.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:21.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:21 smithi038 ceph-mon[29779]: pgmap v647: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-04T16:17:21.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:21 smithi038 ceph-mon[38113]: pgmap v647: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-04T16:17:21.494 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:21.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:21 smithi176 ceph-mon[33619]: pgmap v647: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-04T16:17:21.682 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:21.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:21.818 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:21.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:21.956 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:22.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 18 16 39885 39869 138.418 81 0.00271885 0.00720366 2024-03-04T16:17:22.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:17:22.058 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:22.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:22.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:22.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:22.343 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:22.367 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:22.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:22.504 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:22.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:22.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:22.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:22.763 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:22.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:22.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:22.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:22.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:22.991 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:23.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 19 16 40865 40849 134.356 61.25 0.0123293 0.00742429 2024-03-04T16:17:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.014 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:23.115 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:23.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:23.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:23.285 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:23.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:23.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:23.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:23.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:23.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[29779]: pgmap v648: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3353285729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[29779]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:23.607 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[38113]: pgmap v648: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3353285729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:23 smithi038 ceph-mon[38113]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-mon[33619]: pgmap v648: 148 pgs: 148 active+clean; 2.1 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s 2024-03-04T16:17:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:17:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3353285729' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-mon[33619]: from='client.35618 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:17:23.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:23.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:23.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:23.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:23.816 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:23.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:23.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:23.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:23.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:23.991 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-04T16:17:24.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:17:24.002286+0000 min lat: 0.00125439 max lat: 0.218493 avg lat: 0.00755467 2024-03-04T16:17:24.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:17:24.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 20 16 42338 42322 132.241 92.0625 0.00787592 0.00755467 2024-03-04T16:17:24.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:24.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:24.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:17:24.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:24.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:24.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:24 smithi176 ceph-mon[33619]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:17:24.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:24 smithi176 ceph-mon[33619]: osdmap e306: 8 total, 8 up, 6 in 2024-03-04T16:17:24.397 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:24.499 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:24.536 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:24.580 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:17:24.638 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:24.647 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:24.647 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[29779]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:17:24.647 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[29779]: osdmap e306: 8 total, 8 up, 6 in 2024-03-04T16:17:24.647 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:24.648 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[38113]: from='client.35618 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:17:24.648 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:24 smithi038 ceph-mon[38113]: osdmap e306: 8 total, 8 up, 6 in 2024-03-04T16:17:24.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:24.719 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T16:17:24.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:17:24.752 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:24.778 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:24.813 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:24.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:24.964 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:24.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:25.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 21 16 44065 44049 131.083 107.938 0.00993598 0.00762283 2024-03-04T16:17:25.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:25.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:25.263 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:25.315 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:17:25.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:25.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: pgmap v649: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[29779]: osdmap e307: 8 total, 8 up, 6 in 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: pgmap v649: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:25.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:25.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:25 smithi038 ceph-mon[38113]: osdmap e307: 8 total, 8 up, 6 in 2024-03-04T16:17:25.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:25.545 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: pgmap v649: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2479224237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:25.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:25 smithi176 ceph-mon[33619]: osdmap e307: 8 total, 8 up, 6 in 2024-03-04T16:17:25.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:25.683 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:25.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:25.820 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:25.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:25.960 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:25.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:26.001 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 22 16 45879 45863 130.278 113.375 0.00636693 0.00767089 2024-03-04T16:17:26.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:26.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:26.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:26.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:26.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:26.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:26.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:26.474 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:26.514 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:26.522 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:17:26.523 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:17:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:26 smithi176 ceph-mon[33619]: pgmap v652: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-04T16:17:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:26 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:26.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:26 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:26.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:26.654 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:26.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[29779]: pgmap v652: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-04T16:17:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[38113]: pgmap v652: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-04T16:17:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:26 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:26.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:26.793 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:26.896 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:26.934 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:26.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:26.999 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:27.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 23 15 47623 47608 129.355 109.062 0.00643552 0.00772438 2024-03-04T16:17:27.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:27.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:27.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:27.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:17:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:27.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:27.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:27.373 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:27.440 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:27 smithi038 ceph-mon[29779]: osdmap e308: 8 total, 8 up, 6 in 2024-03-04T16:17:27.442 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:27 smithi038 ceph-mon[38113]: osdmap e308: 8 total, 8 up, 6 in 2024-03-04T16:17:27.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:27.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:27.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:27 smithi176 ceph-mon[33619]: osdmap e308: 8 total, 8 up, 6 in 2024-03-04T16:17:27.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:27.648 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:27.750 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:27.790 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:27.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:27.929 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:27.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:28.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 24 16 49253 49237 128.207 101.812 0.00966681 0.00779476 2024-03-04T16:17:28.020 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:17:28.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:28.062 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:17:28.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:28.101 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:17:28.142 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:17:28.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:28.190 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:17:28.219 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:28.230 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:17:28.276 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:17:28.315 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:17:28.320 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:17:28.321 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:17:28.321 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.362 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:28.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:28.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:28.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:17:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:28.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:28.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:28.639 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:28.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:28 smithi038 ceph-mon[29779]: pgmap v654: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-04T16:17:28.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:28 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3930478908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:28.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:28 smithi038 ceph-mon[38113]: pgmap v654: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-04T16:17:28.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:28 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3930478908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:28.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:28.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:28 smithi176 ceph-mon[33619]: pgmap v654: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-04T16:17:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:28 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3930478908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:28.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:28.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:28.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:28.998 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:29.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 25 16 50832 50816 127.026 98.6875 0.0132021 0.0078669 2024-03-04T16:17:29.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:29.092 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:29.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:29.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:29.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.370 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:29.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:29.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:29.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:29.730 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:29.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:29.848 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:29.874 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:29.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:29.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:30.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 26 16 52389 52373 125.883 97.3125 0.00634262 0.00793933 2024-03-04T16:17:30.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.056 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:30.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:30.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:30.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:30.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:30.594 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:30.633 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:30.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.776 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:30.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:30.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:30.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:30.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[29779]: pgmap v655: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[29779]: osdmap e309: 8 total, 8 up, 6 in 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[38113]: pgmap v655: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:30 smithi038 ceph-mon[38113]: osdmap e309: 8 total, 8 up, 6 in 2024-03-04T16:17:31.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 27 16 54093 54077 125.164 106.5 0.00621146 0.00798492 2024-03-04T16:17:31.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:31.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:30 smithi176 ceph-mon[33619]: pgmap v655: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:17:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:30 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:30 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:30 smithi176 ceph-mon[33619]: osdmap e309: 8 total, 8 up, 6 in 2024-03-04T16:17:31.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:31.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:31.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:31.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:31.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:31.490 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:31.592 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:31.630 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:31.732 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:31.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:31.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:17:31.871 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:31.904 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:31.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:32.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 28 16 55655 55639 124.181 97.625 0.022069 0.0080467 2024-03-04T16:17:32.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:32.017 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:17:32.025 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:17:32.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:32.137 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:32.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:32.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:32.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:32.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:32.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:32.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:32.583 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:32.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:32.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:32.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:32 smithi038 ceph-mon[29779]: pgmap v657: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 813 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:17:32.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:32 smithi038 ceph-mon[29779]: from='client.25594 v1:172.21.15.38:0/210774050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:32.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:32 smithi038 ceph-mon[38113]: pgmap v657: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 813 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:17:32.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:32 smithi038 ceph-mon[38113]: from='client.25594 v1:172.21.15.38:0/210774050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:32.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:32.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:32.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:32.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:33.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 29 16 57209 57193 123.247 97.125 0.0113266 0.00810932 2024-03-04T16:17:33.010 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:32 smithi176 ceph-mon[33619]: pgmap v657: 148 pgs: 148 active+clean; 3.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 813 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-03-04T16:17:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:32 smithi176 ceph-mon[33619]: from='client.25594 v1:172.21.15.38:0/210774050' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:33.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:33.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:33.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:33.290 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:33.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:33.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:33.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:33.562 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:33.567 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:33.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:33.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:33.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.697 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:33.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:33.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:33.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:33.973 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:33.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:33.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:33.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:33.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:34.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 30 16 58644 58628 122.128 89.6875 0.0126208 0.00818287 2024-03-04T16:17:34.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:34.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:34.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:34.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:34.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:34.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:34.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:34.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:34.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.671 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:34.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:34.772 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:34.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:34.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:34.951 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:34.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:34.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:34.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:34 smithi038 ceph-mon[29779]: pgmap v658: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:34.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:34 smithi038 ceph-mon[38113]: pgmap v658: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:35.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 31 16 60213 60197 121.352 98.0625 0.0136715 0.00823569 2024-03-04T16:17:35.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.077 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:35.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:35.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:34 smithi176 ceph-mon[33619]: pgmap v658: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:17:35.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:35.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:35.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.344 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:35.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:35.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:35.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:35.724 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:35.759 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:35.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:35.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:35.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:36.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:36.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 32 15 61594 61579 120.259 86.375 0.0145348 0.00831057 2024-03-04T16:17:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.039 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:36.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:36.140 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:36.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.177 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:36.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:36.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:36.312 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:36.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:36.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:36.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:36.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:36.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:36 smithi176 ceph-mon[33619]: osdmap e310: 8 total, 8 up, 6 in 2024-03-04T16:17:36.683 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:17:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:17:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:62078 10.d e310/296 10:b6e895a0:::benchmark_data_smithi038_155391_object62077:head v 310'3886, mlcod=310'3884)", 2024-03-04T16:17:36.683 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.326131+0000", 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0026307079999999998, 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026700249999999999, 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326131+0000", 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326131+0000", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326132+0000", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3699999999999999e-07 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326190+0000", 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7524e-05 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326201+0000", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1146e-05 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326207+0000", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9410000000000004e-06 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326258+0000", 2024-03-04T16:17:36.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1374e-05 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326280+0000", 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1889e-05 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:62073 10.1 e310/296 10:80f41c39:::benchmark_data_smithi038_155391_object62072:head v 310'3869, mlcod=310'3867)", 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.326221+0000", 2024-03-04T16:17:36.687 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0025403159999999999, 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026496710000000001, 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326221+0000", 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326221+0000", 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326222+0000", 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9999999999999998e-07 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326302+0000", 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0356000000000001e-05 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326313+0000", 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0158e-05 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326317+0000", 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4449999999999998e-06 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326382+0000", 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5298999999999999e-05 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.690 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326445+0000", 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2471999999999994e-05 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:62079 10.5 10:aa57b942:::benchmark_data_smithi038_155391_object62078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.326604+0000", 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0021580850000000001, 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023368310000000002, 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:17:36.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 62079 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326604+0000", 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326604+0000", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326606+0000", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1019999999999999e-06 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326665+0000", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9154000000000003e-05 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326698+0000", 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2709000000000001e-05 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326705+0000", 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5839999999999997e-06 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326762+0000", 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6644999999999997e-05 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326921+0000", 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000158741 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.327134+0000", 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021288700000000001 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:62081 10.5 10.f1425fc5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-03-04T16:17:36.695 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.326809+0000", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0019530400000000001, 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022155159999999998, 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 62081 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326809+0000", 2024-03-04T16:17:36.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326809+0000", 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326810+0000", 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1480000000000001e-06 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326833+0000", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2662000000000001e-05 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326843+0000", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0607e-05 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.326851+0000", 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4159999999999998e-06 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.698 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:62082 10.2 e310/296)", 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.328082+0000", 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00067973299999999996, 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00099775800000000006, 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328082+0000", 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328082+0000", 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328084+0000", 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4389999999999999e-06 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328108+0000", 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4023e-05 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328121+0000", 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.383e-05 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328129+0000", 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2169999999999997e-06 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.701 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:62075 10.6 e310/296)", 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.327514+0000", 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00124825, 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001614954, 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.327514+0000", 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.327514+0000", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.327515+0000", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1009999999999999e-06 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.327578+0000", 2024-03-04T16:17:36.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3393999999999996e-05 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.329082+0000", 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001503866 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.329089+0000", 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7920000000000004e-06 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:62072 10.6 e310/296 10:6be2f594:::benchmark_data_smithi038_155391_object62071:head v 310'3922, mlcod=310'3917)", 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:36.324978+0000", 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.0037837109999999999, 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0042009489999999998, 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.324978+0000", 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.324978+0000", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.324979+0000", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2359999999999999e-06 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.325040+0000", 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0813000000000002e-05 2024-03-04T16:17:36.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.325052+0000", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1562999999999999e-05 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.325058+0000", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7209999999999997e-06 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328420+0000", 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0033619230000000002 2024-03-04T16:17:36.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:36.328441+0000", 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0772e-05 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 7 2024-03-04T16:17:36.708 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:17:36.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:36 smithi038 ceph-mon[29779]: osdmap e310: 8 total, 8 up, 6 in 2024-03-04T16:17:36.709 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:36 smithi038 ceph-mon[38113]: osdmap e310: 8 total, 8 up, 6 in 2024-03-04T16:17:36.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:36.748 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:36.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:36.890 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:36.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:36.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:37.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 33 16 62983 62967 119.243 86.75 0.0353644 0.00837578 2024-03-04T16:17:37.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:37.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:37.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:37.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:37.305 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:37.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:37.443 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:37.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:37.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:37.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:37 smithi176 ceph-mon[33619]: pgmap v660: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-04T16:17:37.617 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:37 smithi038 ceph-mon[29779]: pgmap v660: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-04T16:17:37.618 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:37 smithi038 ceph-mon[38113]: pgmap v660: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-03-04T16:17:37.683 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:37.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:37.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:37.856 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:37.958 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:37.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:37.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:38.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 34 16 64491 64475 118.508 94.25 0.0104088 0.00843354 2024-03-04T16:17:38.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:38.110 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-03-04T16:17:31.604172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3248953694,"num_objects":49764,"num_object_clones":0,"num_object_copies":149292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49764,"num_whiteouts":0,"num_read":2225,"num_read_kb":2071,"num_write":50467,"num_write_kb":3174685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50533,"ondisk_log_size":50533,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":13166500,"kb_used_data":9367092,"kb_used_omap":9,"kb_used_meta":3799350,"kb_avail":549304412,"statfs":{"total":575970213888,"available":562487717888,"internally_reserved":0,"allocated":9591902208,"data_stored":9589045992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":3890535079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":546045952,"num_objects":8332,"num_object_clones":0,"num_object_copies":24996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8332,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8332,"num_write_kb":533248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.296067"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345615+0000","last_change":"2024-03-04T16:16:48.504644+0000","last_active":"2024-03-04T16:17:26.345615+0000","last_peered":"2024-03-04T16:17:26.345615+0000","last_clean":"2024-03-04T16:17:26.345615+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:17:26.345615+0000","last_undegraded":"2024-03-04T16:17:26.345615+0000","last_fullsized":"2024-03-04T16:17:26.345615+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:48.504543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:16:48.504543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.379284+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":529,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351434+0000","last_change":"2024-03-04T16:14:56.671516+0000","last_active":"2024-03-04T16:17:26.351434+0000","last_peered":"2024-03-04T16:17:26.351434+0000","last_clean":"2024-03-04T16:17:26.351434+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:17:26.351434+0000","last_undegraded":"2024-03-04T16:17:26.351434+0000","last_fullsized":"2024-03-04T16:17:26.351434+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:56.671387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:14:56.671387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":393,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352511+0000","last_change":"2024-03-04T16:15:44.274272+0000","last_active":"2024-03-04T16:17:26.352511+0000","last_peered":"2024-03-04T16:17:26.352511+0000","last_clean":"2024-03-04T16:17:26.352511+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:17:26.352511+0000","last_undegraded":"2024-03-04T16:17:26.352511+0000","last_fullsized":"2024-03-04T16:17:26.352511+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:44.274133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:44.274133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.789345+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":582,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346114+0000","last_change":"2024-03-04T16:17:07.218989+0000","last_active":"2024-03-04T16:17:26.346114+0000","last_peered":"2024-03-04T16:17:26.346114+0000","last_clean":"2024-03-04T16:17:26.346114+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:17:26.346114+0000","last_undegraded":"2024-03-04T16:17:26.346114+0000","last_fullsized":"2024-03-04T16:17:26.346114+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:17:07.218848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.218848+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.217883+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":530,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350296+0000","last_change":"2024-03-04T16:16:55.868681+0000","last_active":"2024-03-04T16:17:26.350296+0000","last_peered":"2024-03-04T16:17:26.350296+0000","last_clean":"2024-03-04T16:17:26.350296+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:17:26.350296+0000","last_undegraded":"2024-03-04T16:17:26.350296+0000","last_fullsized":"2024-03-04T16:17:26.350296+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.868581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.868581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:24.049066+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346501+0000","last_change":"2024-03-04T16:15:44.517042+0000","last_active":"2024-03-04T16:17:26.346501+0000","last_peered":"2024-03-04T16:17:26.346501+0000","last_clean":"2024-03-04T16:17:26.346501+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:17:26.346501+0000","last_undegraded":"2024-03-04T16:17:26.346501+0000","last_fullsized":"2024-03-04T16:17:26.346501+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:44.516926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:44.516926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293673+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349425+0000","last_change":"2024-03-04T16:16:07.030698+0000","last_active":"2024-03-04T16:17:26.349425+0000","last_peered":"2024-03-04T16:17:26.349425+0000","last_clean":"2024-03-04T16:17:26.349425+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:17:26.349425+0000","last_undegraded":"2024-03-04T16:17:26.349425+0000","last_fullsized":"2024-03-04T16:17:26.349425+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.030580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.030580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":642,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348672+0000","last_change":"2024-03-04T16:15:51.042206+0000","last_active":"2024-03-04T16:17:26.348672+0000","last_peered":"2024-03-04T16:17:26.348672+0000","last_clean":"2024-03-04T16:17:26.348672+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:17:26.348672+0000","last_undegraded":"2024-03-04T16:17:26.348672+0000","last_fullsized":"2024-03-04T16:17:26.348672+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:15:51.042010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.042010+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":389,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352366+0000","last_change":"2024-03-04T16:16:00.432293+0000","last_active":"2024-03-04T16:17:26.352366+0000","last_peered":"2024-03-04T16:17:26.352366+0000","last_clean":"2024-03-04T16:17:26.352366+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:17:26.352366+0000","last_undegraded":"2024-03-04T16:17:26.352366+0000","last_fullsized":"2024-03-04T16:17:26.352366+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.432170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.432170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.783171+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352738+0000","last_change":"2024-03-04T16:17:03.584844+0000","last_active":"2024-03-04T16:17:26.352738+0000","last_peered":"2024-03-04T16:17:26.352738+0000","last_clean":"2024-03-04T16:17:26.352738+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:17:26.352738+0000","last_undegraded":"2024-03-04T16:17:26.352738+0000","last_fullsized":"2024-03-04T16:17:26.352738+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.584766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.584766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:14.913502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":557,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352362+0000","last_change":"2024-03-04T16:16:18.732834+0000","last_active":"2024-03-04T16:17:26.352362+0000","last_peered":"2024-03-04T16:17:26.352362+0000","last_clean":"2024-03-04T16:17:26.352362+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:17:26.352362+0000","last_undegraded":"2024-03-04T16:17:26.352362+0000","last_fullsized":"2024-03-04T16:17:26.352362+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.732730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.732730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":593,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345826+0000","last_change":"2024-03-04T16:15:00.710265+0000","last_active":"2024-03-04T16:17:26.345826+0000","last_peered":"2024-03-04T16:17:26.345826+0000","last_clean":"2024-03-04T16:17:26.345826+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:17:26.345826+0000","last_undegraded":"2024-03-04T16:17:26.345826+0000","last_fullsized":"2024-03-04T16:17:26.345826+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:00.710134+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:15:00.710134+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.292470+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":540,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.355421+0000","last_change":"2024-03-04T16:16:09.872300+0000","last_active":"2024-03-04T16:17:26.355421+0000","last_peered":"2024-03-04T16:17:26.355421+0000","last_clean":"2024-03-04T16:17:26.355421+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:17:26.355421+0000","last_undegraded":"2024-03-04T16:17:26.355421+0000","last_fullsized":"2024-03-04T16:17:26.355421+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:16:09.872152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348053+0000","last_change":"2024-03-04T16:16:49.431749+0000","last_active":"2024-03-04T16:17:26.348053+0000","last_peered":"2024-03-04T16:17:26.348053+0000","last_clean":"2024-03-04T16:17:26.348053+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:17:26.348053+0000","last_undegraded":"2024-03-04T16:17:26.348053+0000","last_fullsized":"2024-03-04T16:17:26.348053+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:49.431643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.431643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:54.959231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":543,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351656+0000","last_change":"2024-03-04T16:16:06.843357+0000","last_active":"2024-03-04T16:17:26.351656+0000","last_peered":"2024-03-04T16:17:26.351656+0000","last_clean":"2024-03-04T16:17:26.351656+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:17:26.351656+0000","last_undegraded":"2024-03-04T16:17:26.351656+0000","last_fullsized":"2024-03-04T16:17:26.351656+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:06.843231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:06.843231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:30.437526+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":571,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346382+0000","last_change":"2024-03-04T16:15:07.876478+0000","last_active":"2024-03-04T16:17:26.346382+0000","last_peered":"2024-03-04T16:17:26.346382+0000","last_clean":"2024-03-04T16:17:26.346382+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:17:26.346382+0000","last_undegraded":"2024-03-04T16:17:26.346382+0000","last_fullsized":"2024-03-04T16:17:26.346382+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-03-04T16:15:07.876334+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:15:07.876334+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293515+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347684+0000","last_change":"2024-03-04T16:15:43.560939+0000","last_active":"2024-03-04T16:17:26.347684+0000","last_peered":"2024-03-04T16:17:26.347684+0000","last_clean":"2024-03-04T16:17:26.347684+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:17:26.347684+0000","last_undegraded":"2024-03-04T16:17:26.347684+0000","last_fullsized":"2024-03-04T16:17:26.347684+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:43.560826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:43.560826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293570+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357842+0000","last_change":"2024-03-04T16:15:06.747836+0000","last_active":"2024-03-04T16:17:26.357842+0000","last_peered":"2024-03-04T16:17:26.357842+0000","last_clean":"2024-03-04T16:17:26.357842+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:17:26.357842+0000","last_undegraded":"2024-03-04T16:17:26.357842+0000","last_fullsized":"2024-03-04T16:17:26.357842+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:06.747157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:06.747157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130039+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":380,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353440+0000","last_change":"2024-03-04T16:16:03.471410+0000","last_active":"2024-03-04T16:17:26.353440+0000","last_peered":"2024-03-04T16:17:26.353440+0000","last_clean":"2024-03-04T16:17:26.353440+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:17:26.353440+0000","last_undegraded":"2024-03-04T16:17:26.353440+0000","last_fullsized":"2024-03-04T16:17:26.353440+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.471180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.471180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.789432+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"286'34","reported_seq":357,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.356966+0000","last_change":"2024-03-04T16:16:16.420065+0000","last_active":"2024-03-04T16:17:26.356966+0000","last_peered":"2024-03-04T16:17:26.356966+0000","last_clean":"2024-03-04T16:17:26.356966+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:17:26.356966+0000","last_undegraded":"2024-03-04T16:17:26.356966+0000","last_fullsized":"2024-03-04T16:17:26.356966+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.419895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:26.820111+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":524,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351623+0000","last_change":"2024-03-04T16:14:57.996139+0000","last_active":"2024-03-04T16:17:26.351623+0000","last_peered":"2024-03-04T16:17:26.351623+0000","last_clean":"2024-03-04T16:17:26.351623+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:17:26.351623+0000","last_undegraded":"2024-03-04T16:17:26.351623+0000","last_fullsized":"2024-03-04T16:17:26.351623+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:14:57.996007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347295+0000","last_change":"2024-03-04T16:16:52.414765+0000","last_active":"2024-03-04T16:17:26.347295+0000","last_peered":"2024-03-04T16:17:26.347295+0000","last_clean":"2024-03-04T16:17:26.347295+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:17:26.347295+0000","last_undegraded":"2024-03-04T16:17:26.347295+0000","last_fullsized":"2024-03-04T16:17:26.347295+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.414689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.414689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:53.722924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":550,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345933+0000","last_change":"2024-03-04T16:16:17.538604+0000","last_active":"2024-03-04T16:17:26.345933+0000","last_peered":"2024-03-04T16:17:26.345933+0000","last_clean":"2024-03-04T16:17:26.345933+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:17:26.345933+0000","last_undegraded":"2024-03-04T16:17:26.345933+0000","last_fullsized":"2024-03-04T16:17:26.345933+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:17.538492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.538492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.307831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":572,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349507+0000","last_change":"2024-03-04T16:17:03.702758+0000","last_active":"2024-03-04T16:17:26.349507+0000","last_peered":"2024-03-04T16:17:26.349507+0000","last_clean":"2024-03-04T16:17:26.349507+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:17:26.349507+0000","last_undegraded":"2024-03-04T16:17:26.349507+0000","last_fullsized":"2024-03-04T16:17:26.349507+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:17:03.702661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.702661+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.888478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348978+0000","last_change":"2024-03-04T16:16:08.368380+0000","last_active":"2024-03-04T16:17:26.348978+0000","last_peered":"2024-03-04T16:17:26.348978+0000","last_clean":"2024-03-04T16:17:26.348978+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:17:26.348978+0000","last_undegraded":"2024-03-04T16:17:26.348978+0000","last_fullsized":"2024-03-04T16:17:26.348978+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:16:08.368298+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.368298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:33.169102+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":539,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349398+0000","last_change":"2024-03-04T16:17:08.205355+0000","last_active":"2024-03-04T16:17:26.349398+0000","last_peered":"2024-03-04T16:17:26.349398+0000","last_clean":"2024-03-04T16:17:26.349398+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:17:26.349398+0000","last_undegraded":"2024-03-04T16:17:26.349398+0000","last_fullsized":"2024-03-04T16:17:26.349398+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'10","last_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:17:08.205209+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.803342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351688+0000","last_change":"2024-03-04T16:16:15.395144+0000","last_active":"2024-03-04T16:17:26.351688+0000","last_peered":"2024-03-04T16:17:26.351688+0000","last_clean":"2024-03-04T16:17:26.351688+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:17:26.351688+0000","last_undegraded":"2024-03-04T16:17:26.351688+0000","last_fullsized":"2024-03-04T16:17:26.351688+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:15.395015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.395015+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:35.021306+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":416,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352875+0000","last_change":"2024-03-04T16:16:08.556672+0000","last_active":"2024-03-04T16:17:26.352875+0000","last_peered":"2024-03-04T16:17:26.352875+0000","last_clean":"2024-03-04T16:17:26.352875+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:17:26.352875+0000","last_undegraded":"2024-03-04T16:17:26.352875+0000","last_fullsized":"2024-03-04T16:17:26.352875+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:08.553910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.553910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.783228+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357986+0000","last_change":"2024-03-04T16:17:03.492110+0000","last_active":"2024-03-04T16:17:26.357986+0000","last_peered":"2024-03-04T16:17:26.357986+0000","last_clean":"2024-03-04T16:17:26.357986+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:17:26.357986+0000","last_undegraded":"2024-03-04T16:17:26.357986+0000","last_fullsized":"2024-03-04T16:17:26.357986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:28.381622+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":59,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352370+0000","last_change":"2024-03-04T16:17:00.852789+0000","last_active":"2024-03-04T16:17:26.352370+0000","last_peered":"2024-03-04T16:17:26.352370+0000","last_clean":"2024-03-04T16:17:26.352370+0000","last_became_active":"2024-03-04T16:16:17.961588+0000","last_became_peered":"2024-03-04T16:16:17.961588+0000","last_unstale":"2024-03-04T16:17:26.352370+0000","last_undegraded":"2024-03-04T16:17:26.352370+0000","last_fullsized":"2024-03-04T16:17:26.352370+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:00.852665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.852665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:11.304309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":549,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348569+0000","last_change":"2024-03-04T16:16:08.076532+0000","last_active":"2024-03-04T16:17:26.348569+0000","last_peered":"2024-03-04T16:17:26.348569+0000","last_clean":"2024-03-04T16:17:26.348569+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:17:26.348569+0000","last_undegraded":"2024-03-04T16:17:26.348569+0000","last_fullsized":"2024-03-04T16:17:26.348569+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.076402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:36.423506+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":327,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353617+0000","last_change":"2024-03-04T16:17:06.624302+0000","last_active":"2024-03-04T16:17:26.353617+0000","last_peered":"2024-03-04T16:17:26.353617+0000","last_clean":"2024-03-04T16:17:26.353617+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:17:26.353617+0000","last_undegraded":"2024-03-04T16:17:26.353617+0000","last_fullsized":"2024-03-04T16:17:26.353617+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:06.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.624154+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.959596+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":547,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348145+0000","last_change":"2024-03-04T16:16:19.400006+0000","last_active":"2024-03-04T16:17:26.348145+0000","last_peered":"2024-03-04T16:17:26.348145+0000","last_clean":"2024-03-04T16:17:26.348145+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:17:26.348145+0000","last_undegraded":"2024-03-04T16:17:26.348145+0000","last_fullsized":"2024-03-04T16:17:26.348145+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.399876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.399876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:40.118078+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":256,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345747+0000","last_change":"2024-03-04T16:15:58.573709+0000","last_active":"2024-03-04T16:17:26.345747+0000","last_peered":"2024-03-04T16:17:26.345747+0000","last_clean":"2024-03-04T16:17:26.345747+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:17:26.345747+0000","last_undegraded":"2024-03-04T16:17:26.345747+0000","last_fullsized":"2024-03-04T16:17:26.345747+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.573599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.573599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.296311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":381,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352415+0000","last_change":"2024-03-04T16:15:58.418770+0000","last_active":"2024-03-04T16:17:26.352415+0000","last_peered":"2024-03-04T16:17:26.352415+0000","last_clean":"2024-03-04T16:17:26.352415+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:17:26.352415+0000","last_undegraded":"2024-03-04T16:17:26.352415+0000","last_fullsized":"2024-03-04T16:17:26.352415+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.418646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.789407+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":557,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348656+0000","last_change":"2024-03-04T16:16:59.675333+0000","last_active":"2024-03-04T16:17:26.348656+0000","last_peered":"2024-03-04T16:17:26.348656+0000","last_clean":"2024-03-04T16:17:26.348656+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:17:26.348656+0000","last_undegraded":"2024-03-04T16:17:26.348656+0000","last_fullsized":"2024-03-04T16:17:26.348656+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:16:59.675176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.675176+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:25.451412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350445+0000","last_change":"2024-03-04T16:15:09.840395+0000","last_active":"2024-03-04T16:17:26.350445+0000","last_peered":"2024-03-04T16:17:26.350445+0000","last_clean":"2024-03-04T16:17:26.350445+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:17:26.350445+0000","last_undegraded":"2024-03-04T16:17:26.350445+0000","last_fullsized":"2024-03-04T16:17:26.350445+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:09.840155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:15:09.840155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":543,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348415+0000","last_change":"2024-03-04T16:16:03.398326+0000","last_active":"2024-03-04T16:17:26.348415+0000","last_peered":"2024-03-04T16:17:26.348415+0000","last_clean":"2024-03-04T16:17:26.348415+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:17:26.348415+0000","last_undegraded":"2024-03-04T16:17:26.348415+0000","last_fullsized":"2024-03-04T16:17:26.348415+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.398160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.398160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.954572+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":552,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.355595+0000","last_change":"2024-03-04T16:17:01.618230+0000","last_active":"2024-03-04T16:17:26.355595+0000","last_peered":"2024-03-04T16:17:26.355595+0000","last_clean":"2024-03-04T16:17:26.355595+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:17:26.355595+0000","last_undegraded":"2024-03-04T16:17:26.355595+0000","last_fullsized":"2024-03-04T16:17:26.355595+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.618146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.618146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.383599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":574,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348566+0000","last_change":"2024-03-04T16:15:01.756693+0000","last_active":"2024-03-04T16:17:26.348566+0000","last_peered":"2024-03-04T16:17:26.348566+0000","last_clean":"2024-03-04T16:17:26.348566+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:17:26.348566+0000","last_undegraded":"2024-03-04T16:17:26.348566+0000","last_fullsized":"2024-03-04T16:17:26.348566+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:01.756601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:01.756601+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":381,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352508+0000","last_change":"2024-03-04T16:16:16.701023+0000","last_active":"2024-03-04T16:17:26.352508+0000","last_peered":"2024-03-04T16:17:26.352508+0000","last_clean":"2024-03-04T16:17:26.352508+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:17:26.352508+0000","last_undegraded":"2024-03-04T16:17:26.352508+0000","last_fullsized":"2024-03-04T16:17:26.352508+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.700854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.700854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.783143+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":539,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350236+0000","last_change":"2024-03-04T16:16:05.072152+0000","last_active":"2024-03-04T16:17:26.350236+0000","last_peered":"2024-03-04T16:17:26.350236+0000","last_clean":"2024-03-04T16:17:26.350236+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:17:26.350236+0000","last_undegraded":"2024-03-04T16:17:26.350236+0000","last_fullsized":"2024-03-04T16:17:26.350236+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:05.071956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:05.071956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":545,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349898+0000","last_change":"2024-03-04T16:15:49.352442+0000","last_active":"2024-03-04T16:17:26.349898+0000","last_peered":"2024-03-04T16:17:26.349898+0000","last_clean":"2024-03-04T16:17:26.349898+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:17:26.349898+0000","last_undegraded":"2024-03-04T16:17:26.349898+0000","last_fullsized":"2024-03-04T16:17:26.349898+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:49.352334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:49.352334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343571+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":602,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.356928+0000","last_change":"2024-03-04T16:15:49.244892+0000","last_active":"2024-03-04T16:17:26.356928+0000","last_peered":"2024-03-04T16:17:26.356928+0000","last_clean":"2024-03-04T16:17:26.356928+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:17:26.356928+0000","last_undegraded":"2024-03-04T16:17:26.356928+0000","last_fullsized":"2024-03-04T16:17:26.356928+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:49.244760+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:15:49.244760+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130071+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":529,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348197+0000","last_change":"2024-03-04T16:16:00.082788+0000","last_active":"2024-03-04T16:17:26.348197+0000","last_peered":"2024-03-04T16:17:26.348197+0000","last_clean":"2024-03-04T16:17:26.348197+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:17:26.348197+0000","last_undegraded":"2024-03-04T16:17:26.348197+0000","last_fullsized":"2024-03-04T16:17:26.348197+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.082581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":537,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.356482+0000","last_change":"2024-03-04T16:16:10.363294+0000","last_active":"2024-03-04T16:17:26.356482+0000","last_peered":"2024-03-04T16:17:26.356482+0000","last_clean":"2024-03-04T16:17:26.356482+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:17:26.356482+0000","last_undegraded":"2024-03-04T16:17:26.356482+0000","last_fullsized":"2024-03-04T16:17:26.356482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.363176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.363176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130097+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":551,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347041+0000","last_change":"2024-03-04T16:16:57.350376+0000","last_active":"2024-03-04T16:17:26.347041+0000","last_peered":"2024-03-04T16:17:26.347041+0000","last_clean":"2024-03-04T16:17:26.347041+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:17:26.347041+0000","last_undegraded":"2024-03-04T16:17:26.347041+0000","last_fullsized":"2024-03-04T16:17:26.347041+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.350240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.350240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.159288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":590,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346236+0000","last_change":"2024-03-04T16:15:56.553897+0000","last_active":"2024-03-04T16:17:26.346236+0000","last_peered":"2024-03-04T16:17:26.346236+0000","last_clean":"2024-03-04T16:17:26.346236+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:17:26.346236+0000","last_undegraded":"2024-03-04T16:17:26.346236+0000","last_fullsized":"2024-03-04T16:17:26.346236+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:56.553745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.553745+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293598+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":414,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349164+0000","last_change":"2024-03-04T16:16:57.407575+0000","last_active":"2024-03-04T16:17:26.349164+0000","last_peered":"2024-03-04T16:17:26.349164+0000","last_clean":"2024-03-04T16:17:26.349164+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:17:26.349164+0000","last_undegraded":"2024-03-04T16:17:26.349164+0000","last_fullsized":"2024-03-04T16:17:26.349164+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.407499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.407499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.214984+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":743,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352468+0000","last_change":"2024-03-04T16:16:59.613637+0000","last_active":"2024-03-04T16:17:26.352468+0000","last_peered":"2024-03-04T16:17:26.352468+0000","last_clean":"2024-03-04T16:17:26.352468+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:17:26.352468+0000","last_undegraded":"2024-03-04T16:17:26.352468+0000","last_fullsized":"2024-03-04T16:17:26.352468+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:16:59.613549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.613549+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:08.905918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":579,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348416+0000","last_change":"2024-03-04T16:16:14.389925+0000","last_active":"2024-03-04T16:17:26.348416+0000","last_peered":"2024-03-04T16:17:26.348416+0000","last_clean":"2024-03-04T16:17:26.348416+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:17:26.348416+0000","last_undegraded":"2024-03-04T16:17:26.348416+0000","last_fullsized":"2024-03-04T16:17:26.348416+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:14.389807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.389807+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:32.835062+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":591,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350901+0000","last_change":"2024-03-04T16:16:53.775738+0000","last_active":"2024-03-04T16:17:26.350901+0000","last_peered":"2024-03-04T16:17:26.350901+0000","last_clean":"2024-03-04T16:17:26.350901+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:17:26.350901+0000","last_undegraded":"2024-03-04T16:17:26.350901+0000","last_fullsized":"2024-03-04T16:17:26.350901+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:53.775609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.775609+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.927920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":531,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349603+0000","last_change":"2024-03-04T16:15:10.975909+0000","last_active":"2024-03-04T16:17:26.349603+0000","last_peered":"2024-03-04T16:17:26.349603+0000","last_clean":"2024-03-04T16:17:26.349603+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:17:26.349603+0000","last_undegraded":"2024-03-04T16:17:26.349603+0000","last_fullsized":"2024-03-04T16:17:26.349603+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:10.975757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:10.975757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.292499+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":717,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352936+0000","last_change":"2024-03-04T16:15:10.811100+0000","last_active":"2024-03-04T16:17:26.352936+0000","last_peered":"2024-03-04T16:17:26.352936+0000","last_clean":"2024-03-04T16:17:26.352936+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:17:26.352936+0000","last_undegraded":"2024-03-04T16:17:26.352936+0000","last_fullsized":"2024-03-04T16:17:26.352936+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:15:10.810975+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:15:10.810975+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":438,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347946+0000","last_change":"2024-03-04T16:16:04.396775+0000","last_active":"2024-03-04T16:17:26.347946+0000","last_peered":"2024-03-04T16:17:26.347946+0000","last_clean":"2024-03-04T16:17:26.347946+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:17:26.347946+0000","last_undegraded":"2024-03-04T16:17:26.347946+0000","last_fullsized":"2024-03-04T16:17:26.347946+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:04.396254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:04.396254+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343520+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":565,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352753+0000","last_change":"2024-03-04T16:15:07.770440+0000","last_active":"2024-03-04T16:17:26.352753+0000","last_peered":"2024-03-04T16:17:26.352753+0000","last_clean":"2024-03-04T16:17:26.352753+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:17:26.352753+0000","last_undegraded":"2024-03-04T16:17:26.352753+0000","last_fullsized":"2024-03-04T16:17:26.352753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:15:07.770201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:07.770201+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":257,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347379+0000","last_change":"2024-03-04T16:14:50.760982+0000","last_active":"2024-03-04T16:17:26.347379+0000","last_peered":"2024-03-04T16:17:26.347379+0000","last_clean":"2024-03-04T16:17:26.347379+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:17:26.347379+0000","last_undegraded":"2024-03-04T16:17:26.347379+0000","last_fullsized":"2024-03-04T16:17:26.347379+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:50.760883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:50.760883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.292400+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":521,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351539+0000","last_change":"2024-03-04T16:14:49.729462+0000","last_active":"2024-03-04T16:17:26.351539+0000","last_peered":"2024-03-04T16:17:26.351539+0000","last_clean":"2024-03-04T16:17:26.351539+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:17:26.351539+0000","last_undegraded":"2024-03-04T16:17:26.351539+0000","last_fullsized":"2024-03-04T16:17:26.351539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:49.729334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:49.729334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":551,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348864+0000","last_change":"2024-03-04T16:16:55.383375+0000","last_active":"2024-03-04T16:17:26.348864+0000","last_peered":"2024-03-04T16:17:26.348864+0000","last_clean":"2024-03-04T16:17:26.348864+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:17:26.348864+0000","last_undegraded":"2024-03-04T16:17:26.348864+0000","last_fullsized":"2024-03-04T16:17:26.348864+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.383292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.383292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.382040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352906+0000","last_change":"2024-03-04T16:16:53.856464+0000","last_active":"2024-03-04T16:17:26.352906+0000","last_peered":"2024-03-04T16:17:26.352906+0000","last_clean":"2024-03-04T16:17:26.352906+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:17:26.352906+0000","last_undegraded":"2024-03-04T16:17:26.352906+0000","last_fullsized":"2024-03-04T16:17:26.352906+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:16:53.856361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.856361+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.016107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":547,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349853+0000","last_change":"2024-03-04T16:16:56.475032+0000","last_active":"2024-03-04T16:17:26.349853+0000","last_peered":"2024-03-04T16:17:26.349853+0000","last_clean":"2024-03-04T16:17:26.349853+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:17:26.349853+0000","last_undegraded":"2024-03-04T16:17:26.349853+0000","last_fullsized":"2024-03-04T16:17:26.349853+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:56.474881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:16:56.474881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.070090+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353492+0000","last_change":"2024-03-04T16:15:56.777453+0000","last_active":"2024-03-04T16:17:26.353492+0000","last_peered":"2024-03-04T16:17:26.353492+0000","last_clean":"2024-03-04T16:17:26.353492+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:17:26.353492+0000","last_undegraded":"2024-03-04T16:17:26.353492+0000","last_fullsized":"2024-03-04T16:17:26.353492+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:56.777347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.777347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.458684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":450,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352718+0000","last_change":"2024-03-04T16:16:51.831340+0000","last_active":"2024-03-04T16:17:26.352718+0000","last_peered":"2024-03-04T16:17:26.352718+0000","last_clean":"2024-03-04T16:17:26.352718+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:17:26.352718+0000","last_undegraded":"2024-03-04T16:17:26.352718+0000","last_fullsized":"2024-03-04T16:17:26.352718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:51.830496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:16:51.830496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.535280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351710+0000","last_change":"2024-03-04T16:14:53.778826+0000","last_active":"2024-03-04T16:17:26.351710+0000","last_peered":"2024-03-04T16:17:26.351710+0000","last_clean":"2024-03-04T16:17:26.351710+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:17:26.351710+0000","last_undegraded":"2024-03-04T16:17:26.351710+0000","last_fullsized":"2024-03-04T16:17:26.351710+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:14:53.778693+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.778693+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"308'3113","reported_seq":3146,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.960956+0000","last_change":"2024-03-04T16:16:53.660438+0000","last_active":"2024-03-04T16:17:27.960956+0000","last_peered":"2024-03-04T16:17:27.960956+0000","last_clean":"2024-03-04T16:17:27.960956+0000","last_became_active":"2024-03-04T16:16:53.660156+0000","last_became_peered":"2024-03-04T16:16:53.660156+0000","last_unstale":"2024-03-04T16:17:27.960956+0000","last_undegraded":"2024-03-04T16:17:27.960956+0000","last_fullsized":"2024-03-04T16:17:27.960956+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3113,"log_dups_size":0,"ondisk_log_size":3113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.297306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204013568,"num_objects":3113,"num_object_clones":0,"num_object_copies":9339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3113,"num_write_kb":199232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":538,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353247+0000","last_change":"2024-03-04T16:15:48.981473+0000","last_active":"2024-03-04T16:17:26.353247+0000","last_peered":"2024-03-04T16:17:26.353247+0000","last_clean":"2024-03-04T16:17:26.353247+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:17:26.353247+0000","last_undegraded":"2024-03-04T16:17:26.353247+0000","last_fullsized":"2024-03-04T16:17:26.353247+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:48.981357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.981357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":589,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350034+0000","last_change":"2024-03-04T16:17:07.251362+0000","last_active":"2024-03-04T16:17:26.350034+0000","last_peered":"2024-03-04T16:17:26.350034+0000","last_clean":"2024-03-04T16:17:26.350034+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:17:26.350034+0000","last_undegraded":"2024-03-04T16:17:26.350034+0000","last_fullsized":"2024-03-04T16:17:26.350034+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:17:07.251155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.251155+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.475039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":605,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353018+0000","last_change":"2024-03-04T16:14:45.845126+0000","last_active":"2024-03-04T16:17:26.353018+0000","last_peered":"2024-03-04T16:17:26.353018+0000","last_clean":"2024-03-04T16:17:26.353018+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:17:26.353018+0000","last_undegraded":"2024-03-04T16:17:26.353018+0000","last_fullsized":"2024-03-04T16:17:26.353018+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:14:45.844973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.844973+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":740,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349115+0000","last_change":"2024-03-04T16:17:09.268256+0000","last_active":"2024-03-04T16:17:26.349115+0000","last_peered":"2024-03-04T16:17:26.349115+0000","last_clean":"2024-03-04T16:17:26.349115+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:17:26.349115+0000","last_undegraded":"2024-03-04T16:17:26.349115+0000","last_fullsized":"2024-03-04T16:17:26.349115+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:17:09.268137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:09.268137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.542889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"308'3128","reported_seq":3162,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.826845+0000","last_change":"2024-03-04T16:16:53.663167+0000","last_active":"2024-03-04T16:17:28.826845+0000","last_peered":"2024-03-04T16:17:28.826845+0000","last_clean":"2024-03-04T16:17:28.826845+0000","last_became_active":"2024-03-04T16:16:53.661370+0000","last_became_peered":"2024-03-04T16:16:53.661370+0000","last_unstale":"2024-03-04T16:17:28.826845+0000","last_undegraded":"2024-03-04T16:17:28.826845+0000","last_fullsized":"2024-03-04T16:17:28.826845+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3128,"log_dups_size":0,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.342888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204996608,"num_objects":3128,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3128,"num_write_kb":200192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":525,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350399+0000","last_change":"2024-03-04T16:16:18.274771+0000","last_active":"2024-03-04T16:17:26.350399+0000","last_peered":"2024-03-04T16:17:26.350399+0000","last_clean":"2024-03-04T16:17:26.350399+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:17:26.350399+0000","last_undegraded":"2024-03-04T16:17:26.350399+0000","last_fullsized":"2024-03-04T16:17:26.350399+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.274659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.274659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:36.887139+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":553,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349371+0000","last_change":"2024-03-04T16:16:20.383892+0000","last_active":"2024-03-04T16:17:26.349371+0000","last_peered":"2024-03-04T16:17:26.349371+0000","last_clean":"2024-03-04T16:17:26.349371+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:17:26.349371+0000","last_undegraded":"2024-03-04T16:17:26.349371+0000","last_fullsized":"2024-03-04T16:17:26.349371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.383782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.383782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.706926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":573,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346225+0000","last_change":"2024-03-04T16:16:08.552789+0000","last_active":"2024-03-04T16:17:26.346225+0000","last_peered":"2024-03-04T16:17:26.346225+0000","last_clean":"2024-03-04T16:17:26.346225+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:17:26.346225+0000","last_undegraded":"2024-03-04T16:17:26.346225+0000","last_fullsized":"2024-03-04T16:17:26.346225+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:16:08.552635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.552635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.296344+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348312+0000","last_change":"2024-03-04T16:16:24.044051+0000","last_active":"2024-03-04T16:17:26.348312+0000","last_peered":"2024-03-04T16:17:26.348312+0000","last_clean":"2024-03-04T16:17:26.348312+0000","last_became_active":"2024-03-04T16:16:24.043673+0000","last_became_peered":"2024-03-04T16:16:24.043673+0000","last_unstale":"2024-03-04T16:17:26.348312+0000","last_undegraded":"2024-03-04T16:17:26.348312+0000","last_fullsized":"2024-03-04T16:17:26.348312+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:46.262241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:46.262241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"308'3033","reported_seq":3067,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.474782+0000","last_change":"2024-03-04T16:16:53.660736+0000","last_active":"2024-03-04T16:17:28.474782+0000","last_peered":"2024-03-04T16:17:28.474782+0000","last_clean":"2024-03-04T16:17:28.474782+0000","last_became_active":"2024-03-04T16:16:53.660464+0000","last_became_peered":"2024-03-04T16:16:53.660464+0000","last_unstale":"2024-03-04T16:17:28.474782+0000","last_undegraded":"2024-03-04T16:17:28.474782+0000","last_fullsized":"2024-03-04T16:17:28.474782+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3033,"log_dups_size":0,"ondisk_log_size":3033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:01.967586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198770688,"num_objects":3033,"num_object_clones":0,"num_object_copies":9099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3033,"num_write_kb":194112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":545,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347766+0000","last_change":"2024-03-04T16:16:18.374475+0000","last_active":"2024-03-04T16:17:26.347766+0000","last_peered":"2024-03-04T16:17:26.347766+0000","last_clean":"2024-03-04T16:17:26.347766+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:17:26.347766+0000","last_undegraded":"2024-03-04T16:17:26.347766+0000","last_fullsized":"2024-03-04T16:17:26.347766+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.374381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.374381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343597+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":273,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353681+0000","last_change":"2024-03-04T16:16:19.642828+0000","last_active":"2024-03-04T16:17:26.353681+0000","last_peered":"2024-03-04T16:17:26.353681+0000","last_clean":"2024-03-04T16:17:26.353681+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:17:26.353681+0000","last_undegraded":"2024-03-04T16:17:26.353681+0000","last_fullsized":"2024-03-04T16:17:26.353681+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.642722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.642722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.783199+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":551,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350078+0000","last_change":"2024-03-04T16:15:46.375861+0000","last_active":"2024-03-04T16:17:26.350078+0000","last_peered":"2024-03-04T16:17:26.350078+0000","last_clean":"2024-03-04T16:17:26.350078+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:17:26.350078+0000","last_undegraded":"2024-03-04T16:17:26.350078+0000","last_fullsized":"2024-03-04T16:17:26.350078+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:46.375761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:15:46.375761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343490+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":352,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.356311+0000","last_change":"2024-03-04T16:16:00.381546+0000","last_active":"2024-03-04T16:17:26.356311+0000","last_peered":"2024-03-04T16:17:26.356311+0000","last_clean":"2024-03-04T16:17:26.356311+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:17:26.356311+0000","last_undegraded":"2024-03-04T16:17:26.356311+0000","last_fullsized":"2024-03-04T16:17:26.356311+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:16:00.381394+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.381394+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"308'3107","reported_seq":3141,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.491288+0000","last_change":"2024-03-04T16:16:53.660460+0000","last_active":"2024-03-04T16:17:28.491288+0000","last_peered":"2024-03-04T16:17:28.491288+0000","last_clean":"2024-03-04T16:17:28.491288+0000","last_became_active":"2024-03-04T16:16:53.660170+0000","last_became_peered":"2024-03-04T16:16:53.660170+0000","last_unstale":"2024-03-04T16:17:28.491288+0000","last_undegraded":"2024-03-04T16:17:28.491288+0000","last_fullsized":"2024-03-04T16:17:28.491288+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3107,"log_dups_size":0,"ondisk_log_size":3107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.653120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3107,"num_write_kb":198848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348562+0000","last_change":"2024-03-04T16:16:12.401505+0000","last_active":"2024-03-04T16:17:26.348562+0000","last_peered":"2024-03-04T16:17:26.348562+0000","last_clean":"2024-03-04T16:17:26.348562+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:17:26.348562+0000","last_undegraded":"2024-03-04T16:17:26.348562+0000","last_fullsized":"2024-03-04T16:17:26.348562+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.401385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.038102+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":257,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346103+0000","last_change":"2024-03-04T16:14:47.655351+0000","last_active":"2024-03-04T16:17:26.346103+0000","last_peered":"2024-03-04T16:17:26.346103+0000","last_clean":"2024-03-04T16:17:26.346103+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:17:26.346103+0000","last_undegraded":"2024-03-04T16:17:26.346103+0000","last_fullsized":"2024-03-04T16:17:26.346103+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:47.655216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:14:47.655216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":550,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351761+0000","last_change":"2024-03-04T16:15:53.841693+0000","last_active":"2024-03-04T16:17:26.351761+0000","last_peered":"2024-03-04T16:17:26.351761+0000","last_clean":"2024-03-04T16:17:26.351761+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:17:26.351761+0000","last_undegraded":"2024-03-04T16:17:26.351761+0000","last_fullsized":"2024-03-04T16:17:26.351761+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:53.841578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:15:53.841578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353530+0000","last_change":"2024-03-04T16:16:49.762334+0000","last_active":"2024-03-04T16:17:26.353530+0000","last_peered":"2024-03-04T16:17:26.353530+0000","last_clean":"2024-03-04T16:17:26.353530+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:17:26.353530+0000","last_undegraded":"2024-03-04T16:17:26.353530+0000","last_fullsized":"2024-03-04T16:17:26.353530+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:16:49.762239+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.762239+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.560260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"308'3048","reported_seq":3082,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.497113+0000","last_change":"2024-03-04T16:16:53.660429+0000","last_active":"2024-03-04T16:17:28.497113+0000","last_peered":"2024-03-04T16:17:28.497113+0000","last_clean":"2024-03-04T16:17:28.497113+0000","last_became_active":"2024-03-04T16:16:53.660115+0000","last_became_peered":"2024-03-04T16:16:53.660115+0000","last_unstale":"2024-03-04T16:17:28.497113+0000","last_undegraded":"2024-03-04T16:17:28.497113+0000","last_fullsized":"2024-03-04T16:17:28.497113+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3048,"log_dups_size":0,"ondisk_log_size":3048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.638211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199753728,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":195072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":517,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350014+0000","last_change":"2024-03-04T16:14:51.811044+0000","last_active":"2024-03-04T16:17:26.350014+0000","last_peered":"2024-03-04T16:17:26.350014+0000","last_clean":"2024-03-04T16:17:26.350014+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:17:26.350014+0000","last_undegraded":"2024-03-04T16:17:26.350014+0000","last_fullsized":"2024-03-04T16:17:26.350014+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:51.810913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.810913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":566,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352648+0000","last_change":"2024-03-04T16:15:51.894739+0000","last_active":"2024-03-04T16:17:26.352648+0000","last_peered":"2024-03-04T16:17:26.352648+0000","last_clean":"2024-03-04T16:17:26.352648+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:17:26.352648+0000","last_undegraded":"2024-03-04T16:17:26.352648+0000","last_fullsized":"2024-03-04T16:17:26.352648+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:15:51.894452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.894452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":564,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346109+0000","last_change":"2024-03-04T16:15:45.509761+0000","last_active":"2024-03-04T16:17:26.346109+0000","last_peered":"2024-03-04T16:17:26.346109+0000","last_clean":"2024-03-04T16:17:26.346109+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:17:26.346109+0000","last_undegraded":"2024-03-04T16:17:26.346109+0000","last_fullsized":"2024-03-04T16:17:26.346109+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:45.509639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:45.509639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293623+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":841,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357222+0000","last_change":"2024-03-04T16:16:17.407977+0000","last_active":"2024-03-04T16:17:26.357222+0000","last_peered":"2024-03-04T16:17:26.357222+0000","last_clean":"2024-03-04T16:17:26.357222+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:17:26.357222+0000","last_undegraded":"2024-03-04T16:17:26.357222+0000","last_fullsized":"2024-03-04T16:17:26.357222+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:16:17.407775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.407775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:26.763969+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":389,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353566+0000","last_change":"2024-03-04T16:17:01.832332+0000","last_active":"2024-03-04T16:17:26.353566+0000","last_peered":"2024-03-04T16:17:26.353566+0000","last_clean":"2024-03-04T16:17:26.353566+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:17:26.353566+0000","last_undegraded":"2024-03-04T16:17:26.353566+0000","last_fullsized":"2024-03-04T16:17:26.353566+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.832115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.832115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:27.727695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"308'3126","reported_seq":3159,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.952008+0000","last_change":"2024-03-04T16:16:53.657683+0000","last_active":"2024-03-04T16:17:27.952008+0000","last_peered":"2024-03-04T16:17:27.952008+0000","last_clean":"2024-03-04T16:17:27.952008+0000","last_became_active":"2024-03-04T16:16:53.657368+0000","last_became_peered":"2024-03-04T16:16:53.657368+0000","last_unstale":"2024-03-04T16:17:27.952008+0000","last_undegraded":"2024-03-04T16:17:27.952008+0000","last_fullsized":"2024-03-04T16:17:27.952008+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3126,"log_dups_size":0,"ondisk_log_size":3126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:55.076373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3126,"num_write_kb":200064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":540,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350241+0000","last_change":"2024-03-04T16:16:02.471635+0000","last_active":"2024-03-04T16:17:26.350241+0000","last_peered":"2024-03-04T16:17:26.350241+0000","last_clean":"2024-03-04T16:17:26.350241+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:17:26.350241+0000","last_undegraded":"2024-03-04T16:17:26.350241+0000","last_fullsized":"2024-03-04T16:17:26.350241+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.471522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.471522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343546+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":558,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352899+0000","last_change":"2024-03-04T16:16:14.435854+0000","last_active":"2024-03-04T16:17:26.352899+0000","last_peered":"2024-03-04T16:17:26.352899+0000","last_clean":"2024-03-04T16:17:26.352899+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:17:26.352899+0000","last_undegraded":"2024-03-04T16:17:26.352899+0000","last_fullsized":"2024-03-04T16:17:26.352899+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:14.435737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.435737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:41.252474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":266,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348306+0000","last_change":"2024-03-04T16:16:10.021523+0000","last_active":"2024-03-04T16:17:26.348306+0000","last_peered":"2024-03-04T16:17:26.348306+0000","last_clean":"2024-03-04T16:17:26.348306+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:17:26.348306+0000","last_undegraded":"2024-03-04T16:17:26.348306+0000","last_fullsized":"2024-03-04T16:17:26.348306+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.021439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:38.867986+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":189,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.358705+0000","last_change":"2024-03-04T16:16:57.389891+0000","last_active":"2024-03-04T16:17:26.358705+0000","last_peered":"2024-03-04T16:17:26.358705+0000","last_clean":"2024-03-04T16:17:26.358705+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:17:26.358705+0000","last_undegraded":"2024-03-04T16:17:26.358705+0000","last_fullsized":"2024-03-04T16:17:26.358705+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:16:57.389758+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.389758+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.714260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":265,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.355738+0000","last_change":"2024-03-04T16:16:19.696457+0000","last_active":"2024-03-04T16:17:26.355738+0000","last_peered":"2024-03-04T16:17:26.355738+0000","last_clean":"2024-03-04T16:17:26.355738+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:17:26.355738+0000","last_undegraded":"2024-03-04T16:17:26.355738+0000","last_fullsized":"2024-03-04T16:17:26.355738+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.696369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.696369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:39.119521+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"308'3167","reported_seq":3200,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.626378+0000","last_change":"2024-03-04T16:16:53.666174+0000","last_active":"2024-03-04T16:17:28.626378+0000","last_peered":"2024-03-04T16:17:28.626378+0000","last_clean":"2024-03-04T16:17:28.626378+0000","last_became_active":"2024-03-04T16:16:53.665651+0000","last_became_peered":"2024-03-04T16:16:53.665651+0000","last_unstale":"2024-03-04T16:17:28.626378+0000","last_undegraded":"2024-03-04T16:17:28.626378+0000","last_fullsized":"2024-03-04T16:17:28.626378+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3167,"log_dups_size":0,"ondisk_log_size":3167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.126117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207552512,"num_objects":3167,"num_object_clones":0,"num_object_copies":9501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3167,"num_write_kb":202688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353388+0000","last_change":"2024-03-04T16:14:57.859262+0000","last_active":"2024-03-04T16:17:26.353388+0000","last_peered":"2024-03-04T16:17:26.353388+0000","last_clean":"2024-03-04T16:17:26.353388+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:17:26.353388+0000","last_undegraded":"2024-03-04T16:17:26.353388+0000","last_fullsized":"2024-03-04T16:17:26.353388+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:57.859146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:57.859146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":853,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348095+0000","last_change":"2024-03-04T16:16:01.374602+0000","last_active":"2024-03-04T16:17:26.348095+0000","last_peered":"2024-03-04T16:17:26.348095+0000","last_clean":"2024-03-04T16:17:26.348095+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:17:26.348095+0000","last_undegraded":"2024-03-04T16:17:26.348095+0000","last_fullsized":"2024-03-04T16:17:26.348095+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:16:01.374477+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.374477+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.771302+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.354261+0000","last_change":"2024-03-04T16:16:20.104755+0000","last_active":"2024-03-04T16:17:26.354261+0000","last_peered":"2024-03-04T16:17:26.354261+0000","last_clean":"2024-03-04T16:17:26.354261+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:17:26.354261+0000","last_undegraded":"2024-03-04T16:17:26.354261+0000","last_fullsized":"2024-03-04T16:17:26.354261+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.104643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.104643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:35.021593+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349480+0000","last_change":"2024-03-04T16:17:05.303664+0000","last_active":"2024-03-04T16:17:26.349480+0000","last_peered":"2024-03-04T16:17:26.349480+0000","last_clean":"2024-03-04T16:17:26.349480+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:17:26.349480+0000","last_undegraded":"2024-03-04T16:17:26.349480+0000","last_fullsized":"2024-03-04T16:17:26.349480+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:17:05.303531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.303531+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.512468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":552,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349129+0000","last_change":"2024-03-04T16:16:15.349131+0000","last_active":"2024-03-04T16:17:26.349129+0000","last_peered":"2024-03-04T16:17:26.349129+0000","last_clean":"2024-03-04T16:17:26.349129+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:17:26.349129+0000","last_undegraded":"2024-03-04T16:17:26.349129+0000","last_fullsized":"2024-03-04T16:17:26.349129+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:15.348942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.348942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"308'3173","reported_seq":3207,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.663432+0000","last_change":"2024-03-04T16:16:53.660223+0000","last_active":"2024-03-04T16:17:27.663432+0000","last_peered":"2024-03-04T16:17:27.663432+0000","last_clean":"2024-03-04T16:17:27.663432+0000","last_became_active":"2024-03-04T16:16:53.659965+0000","last_became_peered":"2024-03-04T16:16:53.659965+0000","last_unstale":"2024-03-04T16:17:27.663432+0000","last_undegraded":"2024-03-04T16:17:27.663432+0000","last_fullsized":"2024-03-04T16:17:27.663432+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3173,"log_dups_size":0,"ondisk_log_size":3173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.464393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207945728,"num_objects":3173,"num_object_clones":0,"num_object_copies":9519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3173,"num_write_kb":203072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357400+0000","last_change":"2024-03-04T16:16:55.443627+0000","last_active":"2024-03-04T16:17:26.357400+0000","last_peered":"2024-03-04T16:17:26.357400+0000","last_clean":"2024-03-04T16:17:26.357400+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:17:26.357400+0000","last_undegraded":"2024-03-04T16:17:26.357400+0000","last_fullsized":"2024-03-04T16:17:26.357400+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.443547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.443547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.590784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":397,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.354068+0000","last_change":"2024-03-04T16:17:02.856208+0000","last_active":"2024-03-04T16:17:26.354068+0000","last_peered":"2024-03-04T16:17:26.354068+0000","last_clean":"2024-03-04T16:17:26.354068+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:17:26.354068+0000","last_undegraded":"2024-03-04T16:17:26.354068+0000","last_fullsized":"2024-03-04T16:17:26.354068+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:02.856124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:02.856124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:04.756539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1444,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348046+0000","last_change":"2024-03-04T16:16:10.322558+0000","last_active":"2024-03-04T16:17:26.348046+0000","last_peered":"2024-03-04T16:17:26.348046+0000","last_clean":"2024-03-04T16:17:26.348046+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:17:26.348046+0000","last_undegraded":"2024-03-04T16:17:26.348046+0000","last_fullsized":"2024-03-04T16:17:26.348046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:16:10.322412+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.322412+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343626+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":877,"num_read_kb":872,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":433,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352579+0000","last_change":"2024-03-04T16:15:43.291569+0000","last_active":"2024-03-04T16:17:26.352579+0000","last_peered":"2024-03-04T16:17:26.352579+0000","last_clean":"2024-03-04T16:17:26.352579+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:17:26.352579+0000","last_undegraded":"2024-03-04T16:17:26.352579+0000","last_fullsized":"2024-03-04T16:17:26.352579+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:15:43.291436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:43.291436+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.789381+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353773+0000","last_change":"2024-03-04T16:16:13.854634+0000","last_active":"2024-03-04T16:17:26.353773+0000","last_peered":"2024-03-04T16:17:26.353773+0000","last_clean":"2024-03-04T16:17:26.353773+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:17:26.353773+0000","last_undegraded":"2024-03-04T16:17:26.353773+0000","last_fullsized":"2024-03-04T16:17:26.353773+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:13.854523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:13.854523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:36.087819+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"308'3088","reported_seq":3122,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.673512+0000","last_change":"2024-03-04T16:16:53.661446+0000","last_active":"2024-03-04T16:17:27.673512+0000","last_peered":"2024-03-04T16:17:27.673512+0000","last_clean":"2024-03-04T16:17:27.673512+0000","last_became_active":"2024-03-04T16:16:53.661113+0000","last_became_peered":"2024-03-04T16:16:53.661113+0000","last_unstale":"2024-03-04T16:17:27.673512+0000","last_undegraded":"2024-03-04T16:17:27.673512+0000","last_fullsized":"2024-03-04T16:17:27.673512+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3088,"log_dups_size":0,"ondisk_log_size":3088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.556881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202375168,"num_objects":3088,"num_object_clones":0,"num_object_copies":9264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3088,"num_write_kb":197632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351573+0000","last_change":"2024-03-04T16:16:11.401122+0000","last_active":"2024-03-04T16:17:26.351573+0000","last_peered":"2024-03-04T16:17:26.351573+0000","last_clean":"2024-03-04T16:17:26.351573+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:17:26.351573+0000","last_undegraded":"2024-03-04T16:17:26.351573+0000","last_fullsized":"2024-03-04T16:17:26.351573+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.401004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.401004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130214+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357073+0000","last_change":"2024-03-04T16:16:04.339942+0000","last_active":"2024-03-04T16:17:26.357073+0000","last_peered":"2024-03-04T16:17:26.357073+0000","last_clean":"2024-03-04T16:17:26.357073+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:17:26.357073+0000","last_undegraded":"2024-03-04T16:17:26.357073+0000","last_fullsized":"2024-03-04T16:17:26.357073+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:04.339827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:04.339827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":519,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353002+0000","last_change":"2024-03-04T16:14:53.705612+0000","last_active":"2024-03-04T16:17:26.353002+0000","last_peered":"2024-03-04T16:17:26.353002+0000","last_clean":"2024-03-04T16:17:26.353002+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:17:26.353002+0000","last_undegraded":"2024-03-04T16:17:26.353002+0000","last_fullsized":"2024-03-04T16:17:26.353002+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:14:53.705474+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.705474+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":257,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352755+0000","last_change":"2024-03-04T16:16:54.600353+0000","last_active":"2024-03-04T16:17:26.352755+0000","last_peered":"2024-03-04T16:17:26.352755+0000","last_clean":"2024-03-04T16:17:26.352755+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:17:26.352755+0000","last_undegraded":"2024-03-04T16:17:26.352755+0000","last_fullsized":"2024-03-04T16:17:26.352755+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:54.600256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:54.600256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.964184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"308'2971","reported_seq":3005,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.655421+0000","last_change":"2024-03-04T16:16:53.660377+0000","last_active":"2024-03-04T16:17:27.655421+0000","last_peered":"2024-03-04T16:17:27.655421+0000","last_clean":"2024-03-04T16:17:27.655421+0000","last_became_active":"2024-03-04T16:16:53.659725+0000","last_became_peered":"2024-03-04T16:16:53.659725+0000","last_unstale":"2024-03-04T16:17:27.655421+0000","last_undegraded":"2024-03-04T16:17:27.655421+0000","last_fullsized":"2024-03-04T16:17:27.655421+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":2971,"log_dups_size":0,"ondisk_log_size":2971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.365457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194707456,"num_objects":2971,"num_object_clones":0,"num_object_copies":8913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2971,"num_write_kb":190144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":548,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352617+0000","last_change":"2024-03-04T16:15:52.388804+0000","last_active":"2024-03-04T16:17:26.352617+0000","last_peered":"2024-03-04T16:17:26.352617+0000","last_clean":"2024-03-04T16:17:26.352617+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:17:26.352617+0000","last_undegraded":"2024-03-04T16:17:26.352617+0000","last_fullsized":"2024-03-04T16:17:26.352617+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:52.388669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:52.388669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.789458+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":557,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352583+0000","last_change":"2024-03-04T16:16:01.874625+0000","last_active":"2024-03-04T16:17:26.352583+0000","last_peered":"2024-03-04T16:17:26.352583+0000","last_clean":"2024-03-04T16:17:26.352583+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:17:26.352583+0000","last_undegraded":"2024-03-04T16:17:26.352583+0000","last_fullsized":"2024-03-04T16:17:26.352583+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:01.874356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.874356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":577,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.356805+0000","last_change":"2024-03-04T16:15:47.218861+0000","last_active":"2024-03-04T16:17:26.356805+0000","last_peered":"2024-03-04T16:17:26.356805+0000","last_clean":"2024-03-04T16:17:26.356805+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:17:26.356805+0000","last_undegraded":"2024-03-04T16:17:26.356805+0000","last_fullsized":"2024-03-04T16:17:26.356805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:15:47.218155+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:15:47.218155+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130128+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":370,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353222+0000","last_change":"2024-03-04T16:16:11.811999+0000","last_active":"2024-03-04T16:17:26.353222+0000","last_peered":"2024-03-04T16:17:26.353222+0000","last_clean":"2024-03-04T16:17:26.353222+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:17:26.353222+0000","last_undegraded":"2024-03-04T16:17:26.353222+0000","last_fullsized":"2024-03-04T16:17:26.353222+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:16:11.811878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.811878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:33.290211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"308'3139","reported_seq":3173,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.819249+0000","last_change":"2024-03-04T16:16:53.663518+0000","last_active":"2024-03-04T16:17:28.819249+0000","last_peered":"2024-03-04T16:17:28.819249+0000","last_clean":"2024-03-04T16:17:28.819249+0000","last_became_active":"2024-03-04T16:16:53.662075+0000","last_became_peered":"2024-03-04T16:16:53.662075+0000","last_unstale":"2024-03-04T16:17:28.819249+0000","last_undegraded":"2024-03-04T16:17:28.819249+0000","last_fullsized":"2024-03-04T16:17:28.819249+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.663744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205717504,"num_objects":3139,"num_object_clones":0,"num_object_copies":9417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":200896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":554,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348754+0000","last_change":"2024-03-04T16:16:50.462234+0000","last_active":"2024-03-04T16:17:26.348754+0000","last_peered":"2024-03-04T16:17:26.348754+0000","last_clean":"2024-03-04T16:17:26.348754+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:17:26.348754+0000","last_undegraded":"2024-03-04T16:17:26.348754+0000","last_fullsized":"2024-03-04T16:17:26.348754+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:50.462116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:16:50.462116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.011998+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":393,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352810+0000","last_change":"2024-03-04T16:16:07.544810+0000","last_active":"2024-03-04T16:17:26.352810+0000","last_peered":"2024-03-04T16:17:26.352810+0000","last_clean":"2024-03-04T16:17:26.352810+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:17:26.352810+0000","last_undegraded":"2024-03-04T16:17:26.352810+0000","last_fullsized":"2024-03-04T16:17:26.352810+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:07.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.544679+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:26.575787+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":605,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.346003+0000","last_change":"2024-03-04T16:15:48.541576+0000","last_active":"2024-03-04T16:17:26.346003+0000","last_peered":"2024-03-04T16:17:26.346003+0000","last_clean":"2024-03-04T16:17:26.346003+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:17:26.346003+0000","last_undegraded":"2024-03-04T16:17:26.346003+0000","last_fullsized":"2024-03-04T16:17:26.346003+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'35","last_scrub_stamp":"2024-03-04T16:15:48.541142+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.541142+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":715,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348970+0000","last_change":"2024-03-04T16:17:06.654928+0000","last_active":"2024-03-04T16:17:26.348970+0000","last_peered":"2024-03-04T16:17:26.348970+0000","last_clean":"2024-03-04T16:17:26.348970+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:17:26.348970+0000","last_undegraded":"2024-03-04T16:17:26.348970+0000","last_fullsized":"2024-03-04T16:17:26.348970+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:06.654784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.654784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.453080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"308'3073","reported_seq":3107,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.683250+0000","last_change":"2024-03-04T16:16:53.659484+0000","last_active":"2024-03-04T16:17:27.683250+0000","last_peered":"2024-03-04T16:17:27.683250+0000","last_clean":"2024-03-04T16:17:27.683250+0000","last_became_active":"2024-03-04T16:16:53.659122+0000","last_became_peered":"2024-03-04T16:16:53.659122+0000","last_unstale":"2024-03-04T16:17:27.683250+0000","last_undegraded":"2024-03-04T16:17:27.683250+0000","last_fullsized":"2024-03-04T16:17:27.683250+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3073,"log_dups_size":0,"ondisk_log_size":3073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.313098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201392128,"num_objects":3073,"num_object_clones":0,"num_object_copies":9219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3073,"num_write_kb":196672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":538,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345759+0000","last_change":"2024-03-04T16:16:11.572661+0000","last_active":"2024-03-04T16:17:26.345759+0000","last_peered":"2024-03-04T16:17:26.345759+0000","last_clean":"2024-03-04T16:17:26.345759+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:17:26.345759+0000","last_undegraded":"2024-03-04T16:17:26.345759+0000","last_fullsized":"2024-03-04T16:17:26.345759+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.572528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.572528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":541,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351347+0000","last_change":"2024-03-04T16:14:52.759799+0000","last_active":"2024-03-04T16:17:26.351347+0000","last_peered":"2024-03-04T16:17:26.351347+0000","last_clean":"2024-03-04T16:17:26.351347+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:17:26.351347+0000","last_undegraded":"2024-03-04T16:17:26.351347+0000","last_fullsized":"2024-03-04T16:17:26.351347+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:52.759532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:14:52.759532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":632,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348876+0000","last_change":"2024-03-04T16:15:03.777164+0000","last_active":"2024-03-04T16:17:26.348876+0000","last_peered":"2024-03-04T16:17:26.348876+0000","last_clean":"2024-03-04T16:17:26.348876+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:17:26.348876+0000","last_undegraded":"2024-03-04T16:17:26.348876+0000","last_fullsized":"2024-03-04T16:17:26.348876+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-03-04T16:15:03.776866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:03.776866+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":530,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345823+0000","last_change":"2024-03-04T16:14:42.615690+0000","last_active":"2024-03-04T16:17:26.345823+0000","last_peered":"2024-03-04T16:17:26.345823+0000","last_clean":"2024-03-04T16:17:26.345823+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:17:26.345823+0000","last_undegraded":"2024-03-04T16:17:26.345823+0000","last_fullsized":"2024-03-04T16:17:26.345823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:42.615581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:42.615581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.292434+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"308'3052","reported_seq":3086,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.557593+0000","last_change":"2024-03-04T16:16:53.660602+0000","last_active":"2024-03-04T16:17:27.557593+0000","last_peered":"2024-03-04T16:17:27.557593+0000","last_clean":"2024-03-04T16:17:27.557593+0000","last_became_active":"2024-03-04T16:16:53.660328+0000","last_became_peered":"2024-03-04T16:16:53.660328+0000","last_unstale":"2024-03-04T16:17:27.557593+0000","last_undegraded":"2024-03-04T16:17:27.557593+0000","last_fullsized":"2024-03-04T16:17:27.557593+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3052,"log_dups_size":0,"ondisk_log_size":3052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.850706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200015872,"num_objects":3052,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3052,"num_write_kb":195328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.352251+0000","last_change":"2024-03-04T16:14:41.961081+0000","last_active":"2024-03-04T16:17:26.352251+0000","last_peered":"2024-03-04T16:17:26.352251+0000","last_clean":"2024-03-04T16:17:26.352251+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:17:26.352251+0000","last_undegraded":"2024-03-04T16:17:26.352251+0000","last_fullsized":"2024-03-04T16:17:26.352251+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:41.960969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:41.960969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":714,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347723+0000","last_change":"2024-03-04T16:14:59.729903+0000","last_active":"2024-03-04T16:17:26.347723+0000","last_peered":"2024-03-04T16:17:26.347723+0000","last_clean":"2024-03-04T16:17:26.347723+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:17:26.347723+0000","last_undegraded":"2024-03-04T16:17:26.347723+0000","last_fullsized":"2024-03-04T16:17:26.347723+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:14:59.729380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.729380+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":537,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.348798+0000","last_change":"2024-03-04T16:16:15.065620+0000","last_active":"2024-03-04T16:17:26.348798+0000","last_peered":"2024-03-04T16:17:26.348798+0000","last_clean":"2024-03-04T16:17:26.348798+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:17:26.348798+0000","last_undegraded":"2024-03-04T16:17:26.348798+0000","last_fullsized":"2024-03-04T16:17:26.348798+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:15.065428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.065428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.765452+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"301'34","reported_seq":591,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.359162+0000","last_change":"2024-03-04T16:17:05.452594+0000","last_active":"2024-03-04T16:17:26.359162+0000","last_peered":"2024-03-04T16:17:26.359162+0000","last_clean":"2024-03-04T16:17:26.359162+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:17:26.359162+0000","last_undegraded":"2024-03-04T16:17:26.359162+0000","last_fullsized":"2024-03-04T16:17:26.359162+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'34","last_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.452417+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.844380+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"308'3148","reported_seq":3182,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.599133+0000","last_change":"2024-03-04T16:16:53.658520+0000","last_active":"2024-03-04T16:17:28.599133+0000","last_peered":"2024-03-04T16:17:28.599133+0000","last_clean":"2024-03-04T16:17:28.599133+0000","last_became_active":"2024-03-04T16:16:53.658240+0000","last_became_peered":"2024-03-04T16:16:53.658240+0000","last_unstale":"2024-03-04T16:17:28.599133+0000","last_undegraded":"2024-03-04T16:17:28.599133+0000","last_fullsized":"2024-03-04T16:17:28.599133+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3148,"log_dups_size":0,"ondisk_log_size":3148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:09.724216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206307328,"num_objects":3148,"num_object_clones":0,"num_object_copies":9444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3148,"num_write_kb":201472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":528,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350797+0000","last_change":"2024-03-04T16:15:41.965920+0000","last_active":"2024-03-04T16:17:26.350797+0000","last_peered":"2024-03-04T16:17:26.350797+0000","last_clean":"2024-03-04T16:17:26.350797+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:17:26.350797+0000","last_undegraded":"2024-03-04T16:17:26.350797+0000","last_fullsized":"2024-03-04T16:17:26.350797+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:15:41.965768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:41.965768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.353701+0000","last_change":"2024-03-04T16:15:58.806390+0000","last_active":"2024-03-04T16:17:26.353701+0000","last_peered":"2024-03-04T16:17:26.353701+0000","last_clean":"2024-03-04T16:17:26.353701+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:17:26.353701+0000","last_undegraded":"2024-03-04T16:17:26.353701+0000","last_fullsized":"2024-03-04T16:17:26.353701+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.806254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.806254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:28.458654+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":543,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345848+0000","last_change":"2024-03-04T16:16:12.534941+0000","last_active":"2024-03-04T16:17:26.345848+0000","last_peered":"2024-03-04T16:17:26.345848+0000","last_clean":"2024-03-04T16:17:26.345848+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:17:26.345848+0000","last_undegraded":"2024-03-04T16:17:26.345848+0000","last_fullsized":"2024-03-04T16:17:26.345848+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.534849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.534849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.307849+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":595,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347578+0000","last_change":"2024-03-04T16:17:00.348913+0000","last_active":"2024-03-04T16:17:26.347578+0000","last_peered":"2024-03-04T16:17:26.347578+0000","last_clean":"2024-03-04T16:17:26.347578+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:17:26.347578+0000","last_undegraded":"2024-03-04T16:17:26.347578+0000","last_fullsized":"2024-03-04T16:17:26.347578+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:00.348771+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.348771+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.680455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"308'3083","reported_seq":3117,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:27.943121+0000","last_change":"2024-03-04T16:16:53.658266+0000","last_active":"2024-03-04T16:17:27.943121+0000","last_peered":"2024-03-04T16:17:27.943121+0000","last_clean":"2024-03-04T16:17:27.943121+0000","last_became_active":"2024-03-04T16:16:53.657934+0000","last_became_peered":"2024-03-04T16:16:53.657934+0000","last_unstale":"2024-03-04T16:17:27.943121+0000","last_undegraded":"2024-03-04T16:17:27.943121+0000","last_fullsized":"2024-03-04T16:17:27.943121+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3083,"log_dups_size":0,"ondisk_log_size":3083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.803374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202047488,"num_objects":3083,"num_object_clones":0,"num_object_copies":9249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3083,"num_write_kb":197312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":271,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.351207+0000","last_change":"2024-03-04T16:16:16.301977+0000","last_active":"2024-03-04T16:17:26.351207+0000","last_peered":"2024-03-04T16:17:26.351207+0000","last_clean":"2024-03-04T16:17:26.351207+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:17:26.351207+0000","last_undegraded":"2024-03-04T16:17:26.351207+0000","last_fullsized":"2024-03-04T16:17:26.351207+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.301767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.301767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:35.568529+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":537,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.357321+0000","last_change":"2024-03-04T16:16:07.370204+0000","last_active":"2024-03-04T16:17:26.357321+0000","last_peered":"2024-03-04T16:17:26.357321+0000","last_clean":"2024-03-04T16:17:26.357321+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:17:26.357321+0000","last_undegraded":"2024-03-04T16:17:26.357321+0000","last_fullsized":"2024-03-04T16:17:26.357321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.370032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.370032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:31.130186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":257,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.345811+0000","last_change":"2024-03-04T16:16:02.539110+0000","last_active":"2024-03-04T16:17:26.345811+0000","last_peered":"2024-03-04T16:17:26.345811+0000","last_clean":"2024-03-04T16:17:26.345811+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:17:26.345811+0000","last_undegraded":"2024-03-04T16:17:26.345811+0000","last_fullsized":"2024-03-04T16:17:26.345811+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.538977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.538977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293778+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":638,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349323+0000","last_change":"2024-03-04T16:17:01.305039+0000","last_active":"2024-03-04T16:17:26.349323+0000","last_peered":"2024-03-04T16:17:26.349323+0000","last_clean":"2024-03-04T16:17:26.349323+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:17:26.349323+0000","last_undegraded":"2024-03-04T16:17:26.349323+0000","last_fullsized":"2024-03-04T16:17:26.349323+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:17:01.304925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.304925+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.947920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"308'3117","reported_seq":3151,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:28.628209+0000","last_change":"2024-03-04T16:16:53.661912+0000","last_active":"2024-03-04T16:17:28.628209+0000","last_peered":"2024-03-04T16:17:28.628209+0000","last_clean":"2024-03-04T16:17:28.628209+0000","last_became_active":"2024-03-04T16:16:53.661564+0000","last_became_peered":"2024-03-04T16:16:53.661564+0000","last_unstale":"2024-03-04T16:17:28.628209+0000","last_undegraded":"2024-03-04T16:17:28.628209+0000","last_fullsized":"2024-03-04T16:17:28.628209+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3117,"log_dups_size":0,"ondisk_log_size":3117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:53.115414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204275712,"num_objects":3117,"num_object_clones":0,"num_object_copies":9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3117,"num_write_kb":199488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":531,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347787+0000","last_change":"2024-03-04T16:15:47.547158+0000","last_active":"2024-03-04T16:17:26.347787+0000","last_peered":"2024-03-04T16:17:26.347787+0000","last_clean":"2024-03-04T16:17:26.347787+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:17:26.347787+0000","last_undegraded":"2024-03-04T16:17:26.347787+0000","last_fullsized":"2024-03-04T16:17:26.347787+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:47.547022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:47.547022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293648+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":443,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.350032+0000","last_change":"2024-03-04T16:16:07.345839+0000","last_active":"2024-03-04T16:17:26.350032+0000","last_peered":"2024-03-04T16:17:26.350032+0000","last_clean":"2024-03-04T16:17:26.350032+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:17:26.350032+0000","last_undegraded":"2024-03-04T16:17:26.350032+0000","last_fullsized":"2024-03-04T16:17:26.350032+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:16:07.345760+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.345760+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:29.343461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":541,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.349668+0000","last_change":"2024-03-04T16:14:59.774268+0000","last_active":"2024-03-04T16:17:26.349668+0000","last_peered":"2024-03-04T16:17:26.349668+0000","last_clean":"2024-03-04T16:17:26.349668+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:17:26.349668+0000","last_undegraded":"2024-03-04T16:17:26.349668+0000","last_fullsized":"2024-03-04T16:17:26.349668+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.774151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":581,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-03-04T16:17:26.347195+0000","last_change":"2024-03-04T16:15:55.587146+0000","last_active":"2024-03-04T16:17:26.347195+0000","last_peered":"2024-03-04T16:17:26.347195+0000","last_clean":"2024-03-04T16:17:26.347195+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:17:26.347195+0000","last_undegraded":"2024-03-04T16:17:26.347195+0000","last_fullsized":"2024-03-04T16:17:26.347195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:15:55.587052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:55.587052+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:27.293701+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3248357376,"num_objects":49566,"num_object_clones":0,"num_object_copies":148698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49566,"num_write_kb":3172224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9730785280,"data_stored":9730785280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":49566,"ondisk_log_size":49566,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":877,"num_read_kb":872,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070645,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759848,"kb_used_data":1165024,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91985304,"statfs":{"total":95995035648,"available":94192951296,"internally_reserved":0,"allocated":1192984576,"data_stored":1192507538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364985,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442808,"kb_used_data":793840,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92302344,"statfs":{"total":95995035648,"available":94517600256,"internally_reserved":0,"allocated":812892160,"data_stored":812397827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738583,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2487088,"kb_used_data":1950568,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91258064,"statfs":{"total":95995035648,"available":93448257536,"internally_reserved":0,"allocated":1997381632,"data_stored":1996896386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149896,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2396204,"kb_used_data":1584804,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91348948,"statfs":{"total":95995035648,"available":93541322752,"internally_reserved":0,"allocated":1622839296,"data_stored":1622357291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":830863816},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953667,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3042520,"kb_used_data":2506000,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90702632,"statfs":{"total":95995035648,"available":92879495168,"internally_reserved":0,"allocated":2566144000,"data_stored":2565681219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626623,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2038032,"kb_used_data":1366856,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91707120,"statfs":{"total":95995035648,"available":93908090880,"internally_reserved":0,"allocated":1399660544,"data_stored":1399205731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":687274438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2045902848,"data_stored":2045902848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":822411264,"data_stored":822411264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1644560384,"data_stored":1644560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2593587200,"data_stored":2593587200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1420754944,"data_stored":1420754944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1203568640,"data_stored":1203568640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:17:38.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:17:38.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:38.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:38.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:38.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:38.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:38.498 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:38.530 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:38.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:38.632 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:38.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:38.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:38.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:38.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:38.943 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:38.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:38.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:39.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 35 16 65845 65829 117.539 84.625 0.00480978 0.00850295 2024-03-04T16:17:39.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:39.081 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:39.183 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:39.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:39.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:39.351 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:39.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:39.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:39.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:39.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:39 smithi176 ceph-mon[33619]: pgmap v661: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-04T16:17:39.628 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:39.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:39.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:39 smithi038 ceph-mon[29779]: pgmap v661: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-04T16:17:39.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:39 smithi038 ceph-mon[38113]: pgmap v661: 148 pgs: 148 active+clean; 3.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-04T16:17:39.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:39.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:39.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:39.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:40.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 36 16 67242 67226 116.7 87.3125 0.0206328 0.00856453 2024-03-04T16:17:40.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:40.049 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:40.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:17:40.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:40.221 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.258 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:40.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:40.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:40.513 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:40.615 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:40.648 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:40.750 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:40.787 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:40.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:40.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:40.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:40.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:41.002 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 37 16 68609 68593 115.855 85.4375 0.0137044 0.00862641 2024-03-04T16:17:41.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.084 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:41.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:41.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:41.222 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:41.324 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.360 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:41.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:41.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.511 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:41.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:41.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:41 smithi038 ceph-mon[29779]: pgmap v662: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-04T16:17:41.598 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:41 smithi038 ceph-mon[29779]: osdmap e311: 8 total, 8 up, 6 in 2024-03-04T16:17:41.598 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:41 smithi038 ceph-mon[38113]: pgmap v662: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-04T16:17:41.598 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:41 smithi038 ceph-mon[38113]: osdmap e311: 8 total, 8 up, 6 in 2024-03-04T16:17:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:41 smithi176 ceph-mon[33619]: pgmap v662: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 102 MiB/s wr, 1.64k op/s 2024-03-04T16:17:41.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:41 smithi176 ceph-mon[33619]: osdmap e311: 8 total, 8 up, 6 in 2024-03-04T16:17:41.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:41.649 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:41.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:41.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:41.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:41.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:41.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:42.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 38 16 70067 70051 115.204 91.125 0.00571414 0.00867534 2024-03-04T16:17:42.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:42.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:42.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:42.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:42.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:42.223 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:42.294 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:42.308 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:42.341 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:17:42.361 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:42.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:42.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:42 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:17:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:42 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:17:42.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:42.600 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:42 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:17:42.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:42.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.776 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:42.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:42.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:42.916 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:42.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:42.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:43.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 39 16 71530 71514 114.594 91.4375 0.00688376 0.00872148 2024-03-04T16:17:43.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:43.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:43.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:43.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:43.332 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:43.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:43.473 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[29779]: pgmap v664: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[29779]: osdmap e312: 8 total, 8 up, 6 in 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[38113]: pgmap v664: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:17:43.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-mon[38113]: osdmap e312: 8 total, 8 up, 6 in 2024-03-04T16:17:43.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:43.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:43 smithi176 ceph-mon[33619]: pgmap v664: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:17:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:43 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:17:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:43 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1105392574' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:17:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:43 smithi176 ceph-mon[33619]: osdmap e312: 8 total, 8 up, 6 in 2024-03-04T16:17:43.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:43.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:43.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:43.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:43.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:43.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:44.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:17:44.004047+0000 min lat: 0.00125439 max lat: 0.218493 avg lat: 0.0087649 2024-03-04T16:17:44.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:17:44.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 40 16 73004 72988 114.032 92.125 0.0100897 0.0087649 2024-03-04T16:17:44.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:44.044 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:44.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:44.181 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:44.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:44.316 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:44.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:44.439 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:17:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:44.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:44.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:17:44.482 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:44.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:17:44.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:17:44.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:17:44.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:44 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:17:44.523 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:17:44.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:44.572 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:17:44.593 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:44 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:17:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:44 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:17:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:44 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:17:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:44 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:17:44.615 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:17:44.657 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:17:44.695 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:44.697 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:17:44.741 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:17:44.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:17:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:17:44.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:44.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:44.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:45.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 41 16 74382 74366 113.351 86.125 0.00992016 0.00881695 2024-03-04T16:17:45.019 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:45.121 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:45.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:45.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:45.295 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:45.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:45.430 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:45.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:45.569 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: pgmap v666: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:17:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:45 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2077797727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:45.671 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:45.704 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: pgmap v666: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2077797727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: pgmap v666: 148 pgs: 148 active+clean; 4.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:17:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:17:45.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:17:45.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:17:45.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:45 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2077797727' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:45.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.846 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:45.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:45.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:45.982 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:45.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:46.003 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 42 16 75877 75861 112.877 93.4375 0.011453 0.00885429 2024-03-04T16:17:46.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:46.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:46.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:46.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:46.366 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:46.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:46.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:46.538 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:46.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:46.679 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:46.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:17:46.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:17:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[29779]: pgmap v667: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-04T16:17:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:17:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:17:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:46 smithi038 ceph-mon[38113]: pgmap v667: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-04T16:17:46.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:46.820 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:46 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:17:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:46 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:17:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:46 smithi176 ceph-mon[33619]: pgmap v667: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s 2024-03-04T16:17:46.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:46.967 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:46.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:47.004 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 43 16 77347 77331 112.386 91.875 0.00624287 0.00889305 2024-03-04T16:17:47.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:47.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:47.136 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:17:47.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:47.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:47.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:47.403 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:17:47.486 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:17:47.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:47 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:17:47.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:47.541 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:47.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:47.685 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:47.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:47.825 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:47.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:17:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:47 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:17:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:47 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:17:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:47 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:17:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:47 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:17:47.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:47.961 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:47.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:47.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:48.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 44 16 78740 78724 111.81 87.0625 0.00510276 0.00893874 2024-03-04T16:17:48.062 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:48.078 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:17:48.078 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:17:48.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:48.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.239 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:48.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:48.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:48.369 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:48.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:48.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:48.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:48.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:48.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: pgmap v668: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:17:48.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: pgmap v668: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:17:48.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:48 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:17:48.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:48.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: pgmap v668: 148 pgs: 148 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 891 op/s 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:17:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:48 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:17:48.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:48.925 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:48.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:49.004 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 45 16 80127 80111 111.252 86.6875 0.0103974 0.00898358 2024-03-04T16:17:49.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:49.067 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:49.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:49.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:17:49.222 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:49.229 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:49.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:49.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:49.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:49.410 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:49.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:17:49.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:49 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:17:49.497 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:17:49.508 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:17:49.513 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:49.549 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:49.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:49.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:49.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:49.809 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:49.845 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:49.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:49 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:17:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:49 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:17:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:49 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:17:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:49 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:17:49.948 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:49.987 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:49.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:50.004 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 46 16 81426 81410 110.598 81.1875 0.0154872 0.00903729 2024-03-04T16:17:50.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.125 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:50.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:50.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:50.270 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:50.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:50.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.413 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:50.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:50.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.576 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:50.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:50.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: from='client.35651 v1:172.21.15.38:0/2407602436' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[29779]: pgmap v669: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: from='client.35651 v1:172.21.15.38:0/2407602436' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:50 smithi038 ceph-mon[38113]: pgmap v669: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:17:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:50.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:50.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: from='client.35651 v1:172.21.15.38:0/2407602436' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:17:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:50 smithi176 ceph-mon[33619]: pgmap v669: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-04T16:17:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:50.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:50.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:50.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:51.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 47 16 82896 82880 110.199 91.875 0.00792765 0.00906979 2024-03-04T16:17:51.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:51.123 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:51.159 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:51.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:51.297 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-03-04T16:17:47.608473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4712765790,"num_objects":72100,"num_object_clones":0,"num_object_copies":216300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72100,"num_whiteouts":0,"num_read":2240,"num_read_kb":2086,"num_write":72803,"num_write_kb":4604189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66369,"ondisk_log_size":66369,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":17491520,"kb_used_data":13628220,"kb_used_omap":9,"kb_used_meta":3863222,"kb_avail":544979392,"statfs":{"total":575970213888,"available":558058897408,"internally_reserved":0,"allocated":13955297280,"data_stored":13952321222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":3955940007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,5,7,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":467206144,"num_objects":7129,"num_object_clones":0,"num_object_copies":21387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7129,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7129,"num_write_kb":456256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002465"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326000+0000","last_change":"2024-03-04T16:16:48.504644+0000","last_active":"2024-03-04T16:17:42.326000+0000","last_peered":"2024-03-04T16:17:42.326000+0000","last_clean":"2024-03-04T16:17:42.326000+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:17:42.326000+0000","last_undegraded":"2024-03-04T16:17:42.326000+0000","last_fullsized":"2024-03-04T16:17:42.326000+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:48.504543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:16:48.504543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.379284+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.331195+0000","last_change":"2024-03-04T16:14:56.671516+0000","last_active":"2024-03-04T16:17:42.331195+0000","last_peered":"2024-03-04T16:17:42.331195+0000","last_clean":"2024-03-04T16:17:42.331195+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:17:42.331195+0000","last_undegraded":"2024-03-04T16:17:42.331195+0000","last_fullsized":"2024-03-04T16:17:42.331195+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:56.671387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:14:56.671387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284060+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":401,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336275+0000","last_change":"2024-03-04T16:15:44.274272+0000","last_active":"2024-03-04T16:17:42.336275+0000","last_peered":"2024-03-04T16:17:42.336275+0000","last_clean":"2024-03-04T16:17:42.336275+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:17:42.336275+0000","last_undegraded":"2024-03-04T16:17:42.336275+0000","last_fullsized":"2024-03-04T16:17:42.336275+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:44.274133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:44.274133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660356+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":590,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.325642+0000","last_change":"2024-03-04T16:17:07.218989+0000","last_active":"2024-03-04T16:17:42.325642+0000","last_peered":"2024-03-04T16:17:42.325642+0000","last_clean":"2024-03-04T16:17:42.325642+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:17:42.325642+0000","last_undegraded":"2024-03-04T16:17:42.325642+0000","last_fullsized":"2024-03-04T16:17:42.325642+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:17:07.218848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.218848+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.217883+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347384+0000","last_change":"2024-03-04T16:16:55.868681+0000","last_active":"2024-03-04T16:17:42.347384+0000","last_peered":"2024-03-04T16:17:42.347384+0000","last_clean":"2024-03-04T16:17:42.347384+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:17:42.347384+0000","last_undegraded":"2024-03-04T16:17:42.347384+0000","last_fullsized":"2024-03-04T16:17:42.347384+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.868581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.868581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:24.049066+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.325773+0000","last_change":"2024-03-04T16:15:44.517042+0000","last_active":"2024-03-04T16:17:42.325773+0000","last_peered":"2024-03-04T16:17:42.325773+0000","last_clean":"2024-03-04T16:17:42.325773+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:17:42.325773+0000","last_undegraded":"2024-03-04T16:17:42.325773+0000","last_fullsized":"2024-03-04T16:17:42.325773+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:44.516926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:44.516926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.346557+0000","last_change":"2024-03-04T16:16:07.030698+0000","last_active":"2024-03-04T16:17:42.346557+0000","last_peered":"2024-03-04T16:17:42.346557+0000","last_clean":"2024-03-04T16:17:42.346557+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:17:42.346557+0000","last_undegraded":"2024-03-04T16:17:42.346557+0000","last_fullsized":"2024-03-04T16:17:42.346557+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.030580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.030580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190685+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":650,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.344327+0000","last_change":"2024-03-04T16:15:51.042206+0000","last_active":"2024-03-04T16:17:42.344327+0000","last_peered":"2024-03-04T16:17:42.344327+0000","last_clean":"2024-03-04T16:17:42.344327+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:17:42.344327+0000","last_undegraded":"2024-03-04T16:17:42.344327+0000","last_fullsized":"2024-03-04T16:17:42.344327+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:15:51.042010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.042010+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190630+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":397,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332090+0000","last_change":"2024-03-04T16:16:00.432293+0000","last_active":"2024-03-04T16:17:42.332090+0000","last_peered":"2024-03-04T16:17:42.332090+0000","last_clean":"2024-03-04T16:17:42.332090+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:17:42.332090+0000","last_undegraded":"2024-03-04T16:17:42.332090+0000","last_fullsized":"2024-03-04T16:17:42.332090+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.432170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.432170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660458+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335866+0000","last_change":"2024-03-04T16:17:03.584844+0000","last_active":"2024-03-04T16:17:42.335866+0000","last_peered":"2024-03-04T16:17:42.335866+0000","last_clean":"2024-03-04T16:17:42.335866+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:17:42.335866+0000","last_undegraded":"2024-03-04T16:17:42.335866+0000","last_fullsized":"2024-03-04T16:17:42.335866+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.584766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.584766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:14.913502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":565,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336241+0000","last_change":"2024-03-04T16:16:18.732834+0000","last_active":"2024-03-04T16:17:42.336241+0000","last_peered":"2024-03-04T16:17:42.336241+0000","last_clean":"2024-03-04T16:17:42.336241+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:17:42.336241+0000","last_undegraded":"2024-03-04T16:17:42.336241+0000","last_fullsized":"2024-03-04T16:17:42.336241+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.732730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.732730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454143+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":601,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326155+0000","last_change":"2024-03-04T16:15:00.710265+0000","last_active":"2024-03-04T16:17:42.326155+0000","last_peered":"2024-03-04T16:17:42.326155+0000","last_clean":"2024-03-04T16:17:42.326155+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:17:42.326155+0000","last_undegraded":"2024-03-04T16:17:42.326155+0000","last_fullsized":"2024-03-04T16:17:42.326155+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:00.710134+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:15:00.710134+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":548,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336821+0000","last_change":"2024-03-04T16:16:09.872300+0000","last_active":"2024-03-04T16:17:42.336821+0000","last_peered":"2024-03-04T16:17:42.336821+0000","last_clean":"2024-03-04T16:17:42.336821+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:17:42.336821+0000","last_undegraded":"2024-03-04T16:17:42.336821+0000","last_fullsized":"2024-03-04T16:17:42.336821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:16:09.872152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454171+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332864+0000","last_change":"2024-03-04T16:16:49.431749+0000","last_active":"2024-03-04T16:17:42.332864+0000","last_peered":"2024-03-04T16:17:42.332864+0000","last_clean":"2024-03-04T16:17:42.332864+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:17:42.332864+0000","last_undegraded":"2024-03-04T16:17:42.332864+0000","last_fullsized":"2024-03-04T16:17:42.332864+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:49.431643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.431643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:54.959231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335973+0000","last_change":"2024-03-04T16:16:06.843357+0000","last_active":"2024-03-04T16:17:42.335973+0000","last_peered":"2024-03-04T16:17:42.335973+0000","last_clean":"2024-03-04T16:17:42.335973+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:17:42.335973+0000","last_undegraded":"2024-03-04T16:17:42.335973+0000","last_fullsized":"2024-03-04T16:17:42.335973+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:06.843231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:06.843231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":579,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326568+0000","last_change":"2024-03-04T16:15:07.876478+0000","last_active":"2024-03-04T16:17:42.326568+0000","last_peered":"2024-03-04T16:17:42.326568+0000","last_clean":"2024-03-04T16:17:42.326568+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:17:42.326568+0000","last_undegraded":"2024-03-04T16:17:42.326568+0000","last_fullsized":"2024-03-04T16:17:42.326568+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-03-04T16:15:07.876334+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:15:07.876334+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.327182+0000","last_change":"2024-03-04T16:15:43.560939+0000","last_active":"2024-03-04T16:17:42.327182+0000","last_peered":"2024-03-04T16:17:42.327182+0000","last_clean":"2024-03-04T16:17:42.327182+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:17:42.327182+0000","last_undegraded":"2024-03-04T16:17:42.327182+0000","last_fullsized":"2024-03-04T16:17:42.327182+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:43.560826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:43.560826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332353+0000","last_change":"2024-03-04T16:15:06.747836+0000","last_active":"2024-03-04T16:17:42.332353+0000","last_peered":"2024-03-04T16:17:42.332353+0000","last_clean":"2024-03-04T16:17:42.332353+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:17:42.332353+0000","last_undegraded":"2024-03-04T16:17:42.332353+0000","last_fullsized":"2024-03-04T16:17:42.332353+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:06.747157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:06.747157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284092+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":389,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336620+0000","last_change":"2024-03-04T16:16:03.471410+0000","last_active":"2024-03-04T16:17:42.336620+0000","last_peered":"2024-03-04T16:17:42.336620+0000","last_clean":"2024-03-04T16:17:42.336620+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:17:42.336620+0000","last_undegraded":"2024-03-04T16:17:42.336620+0000","last_fullsized":"2024-03-04T16:17:42.336620+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.471180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.471180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"286'34","reported_seq":365,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332241+0000","last_change":"2024-03-04T16:16:16.420065+0000","last_active":"2024-03-04T16:17:42.332241+0000","last_peered":"2024-03-04T16:17:42.332241+0000","last_clean":"2024-03-04T16:17:42.332241+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:17:42.332241+0000","last_undegraded":"2024-03-04T16:17:42.332241+0000","last_fullsized":"2024-03-04T16:17:42.332241+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.419895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284357+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":532,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347083+0000","last_change":"2024-03-04T16:14:57.996139+0000","last_active":"2024-03-04T16:17:42.347083+0000","last_peered":"2024-03-04T16:17:42.347083+0000","last_clean":"2024-03-04T16:17:42.347083+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:17:42.347083+0000","last_undegraded":"2024-03-04T16:17:42.347083+0000","last_fullsized":"2024-03-04T16:17:42.347083+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:14:57.996007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190271+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326260+0000","last_change":"2024-03-04T16:16:52.414765+0000","last_active":"2024-03-04T16:17:42.326260+0000","last_peered":"2024-03-04T16:17:42.326260+0000","last_clean":"2024-03-04T16:17:42.326260+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:17:42.326260+0000","last_undegraded":"2024-03-04T16:17:42.326260+0000","last_fullsized":"2024-03-04T16:17:42.326260+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.414689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.414689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:53.722924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326300+0000","last_change":"2024-03-04T16:16:17.538604+0000","last_active":"2024-03-04T16:17:42.326300+0000","last_peered":"2024-03-04T16:17:42.326300+0000","last_clean":"2024-03-04T16:17:42.326300+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:17:42.326300+0000","last_undegraded":"2024-03-04T16:17:42.326300+0000","last_fullsized":"2024-03-04T16:17:42.326300+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:17.538492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.538492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":580,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.351584+0000","last_change":"2024-03-04T16:17:03.702758+0000","last_active":"2024-03-04T16:17:42.351584+0000","last_peered":"2024-03-04T16:17:42.351584+0000","last_clean":"2024-03-04T16:17:42.351584+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:17:42.351584+0000","last_undegraded":"2024-03-04T16:17:42.351584+0000","last_fullsized":"2024-03-04T16:17:42.351584+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:17:03.702661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.702661+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.888478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336789+0000","last_change":"2024-03-04T16:16:08.368380+0000","last_active":"2024-03-04T16:17:42.336789+0000","last_peered":"2024-03-04T16:17:42.336789+0000","last_clean":"2024-03-04T16:17:42.336789+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:17:42.336789+0000","last_undegraded":"2024-03-04T16:17:42.336789+0000","last_fullsized":"2024-03-04T16:17:42.336789+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:16:08.368298+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.368298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.221866+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326974+0000","last_change":"2024-03-04T16:17:08.205355+0000","last_active":"2024-03-04T16:17:42.326974+0000","last_peered":"2024-03-04T16:17:42.326974+0000","last_clean":"2024-03-04T16:17:42.326974+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:17:42.326974+0000","last_undegraded":"2024-03-04T16:17:42.326974+0000","last_fullsized":"2024-03-04T16:17:42.326974+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'10","last_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:17:08.205209+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.803342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.331787+0000","last_change":"2024-03-04T16:16:15.395144+0000","last_active":"2024-03-04T16:17:42.331787+0000","last_peered":"2024-03-04T16:17:42.331787+0000","last_clean":"2024-03-04T16:17:42.331787+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:17:42.331787+0000","last_undegraded":"2024-03-04T16:17:42.331787+0000","last_fullsized":"2024-03-04T16:17:42.331787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:15.395015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.395015+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":425,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338187+0000","last_change":"2024-03-04T16:16:08.556672+0000","last_active":"2024-03-04T16:17:42.338187+0000","last_peered":"2024-03-04T16:17:42.338187+0000","last_clean":"2024-03-04T16:17:42.338187+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:17:42.338187+0000","last_undegraded":"2024-03-04T16:17:42.338187+0000","last_fullsized":"2024-03-04T16:17:42.338187+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:08.553910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.553910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660589+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332499+0000","last_change":"2024-03-04T16:17:03.492110+0000","last_active":"2024-03-04T16:17:42.332499+0000","last_peered":"2024-03-04T16:17:42.332499+0000","last_clean":"2024-03-04T16:17:42.332499+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:17:42.332499+0000","last_undegraded":"2024-03-04T16:17:42.332499+0000","last_fullsized":"2024-03-04T16:17:42.332499+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:28.381622+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":67,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337843+0000","last_change":"2024-03-04T16:17:00.852789+0000","last_active":"2024-03-04T16:17:42.337843+0000","last_peered":"2024-03-04T16:17:42.337843+0000","last_clean":"2024-03-04T16:17:42.337843+0000","last_became_active":"2024-03-04T16:16:17.961588+0000","last_became_peered":"2024-03-04T16:16:17.961588+0000","last_unstale":"2024-03-04T16:17:42.337843+0000","last_undegraded":"2024-03-04T16:17:42.337843+0000","last_fullsized":"2024-03-04T16:17:42.337843+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:00.852665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.852665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:11.304309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":557,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343516+0000","last_change":"2024-03-04T16:16:08.076532+0000","last_active":"2024-03-04T16:17:42.343516+0000","last_peered":"2024-03-04T16:17:42.343516+0000","last_clean":"2024-03-04T16:17:42.343516+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:17:42.343516+0000","last_undegraded":"2024-03-04T16:17:42.343516+0000","last_fullsized":"2024-03-04T16:17:42.343516+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.076402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190713+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":335,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339673+0000","last_change":"2024-03-04T16:17:06.624302+0000","last_active":"2024-03-04T16:17:42.339673+0000","last_peered":"2024-03-04T16:17:42.339673+0000","last_clean":"2024-03-04T16:17:42.339673+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:17:42.339673+0000","last_undegraded":"2024-03-04T16:17:42.339673+0000","last_fullsized":"2024-03-04T16:17:42.339673+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:06.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.624154+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.959596+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332948+0000","last_change":"2024-03-04T16:16:19.400006+0000","last_active":"2024-03-04T16:17:42.332948+0000","last_peered":"2024-03-04T16:17:42.332948+0000","last_clean":"2024-03-04T16:17:42.332948+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:17:42.332948+0000","last_undegraded":"2024-03-04T16:17:42.332948+0000","last_fullsized":"2024-03-04T16:17:42.332948+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.399876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.399876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.230365+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":265,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.325887+0000","last_change":"2024-03-04T16:15:58.573709+0000","last_active":"2024-03-04T16:17:42.325887+0000","last_peered":"2024-03-04T16:17:42.325887+0000","last_clean":"2024-03-04T16:17:42.325887+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:17:42.325887+0000","last_undegraded":"2024-03-04T16:17:42.325887+0000","last_fullsized":"2024-03-04T16:17:42.325887+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.573599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.573599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":389,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336175+0000","last_change":"2024-03-04T16:15:58.418770+0000","last_active":"2024-03-04T16:17:42.336175+0000","last_peered":"2024-03-04T16:17:42.336175+0000","last_clean":"2024-03-04T16:17:42.336175+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:17:42.336175+0000","last_undegraded":"2024-03-04T16:17:42.336175+0000","last_fullsized":"2024-03-04T16:17:42.336175+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.418646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":565,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343608+0000","last_change":"2024-03-04T16:16:59.675333+0000","last_active":"2024-03-04T16:17:42.343608+0000","last_peered":"2024-03-04T16:17:42.343608+0000","last_clean":"2024-03-04T16:17:42.343608+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:17:42.343608+0000","last_undegraded":"2024-03-04T16:17:42.343608+0000","last_fullsized":"2024-03-04T16:17:42.343608+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:16:59.675176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.675176+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:25.451412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347554+0000","last_change":"2024-03-04T16:15:09.840395+0000","last_active":"2024-03-04T16:17:42.347554+0000","last_peered":"2024-03-04T16:17:42.347554+0000","last_clean":"2024-03-04T16:17:42.347554+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:17:42.347554+0000","last_undegraded":"2024-03-04T16:17:42.347554+0000","last_fullsized":"2024-03-04T16:17:42.347554+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:09.840155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:15:09.840155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190512+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337688+0000","last_change":"2024-03-04T16:16:03.398326+0000","last_active":"2024-03-04T16:17:42.337688+0000","last_peered":"2024-03-04T16:17:42.337688+0000","last_clean":"2024-03-04T16:17:42.337688+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:17:42.337688+0000","last_undegraded":"2024-03-04T16:17:42.337688+0000","last_fullsized":"2024-03-04T16:17:42.337688+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.398160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.398160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.221808+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":560,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338938+0000","last_change":"2024-03-04T16:17:01.618230+0000","last_active":"2024-03-04T16:17:42.338938+0000","last_peered":"2024-03-04T16:17:42.338938+0000","last_clean":"2024-03-04T16:17:42.338938+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:17:42.338938+0000","last_undegraded":"2024-03-04T16:17:42.338938+0000","last_fullsized":"2024-03-04T16:17:42.338938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.618146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.618146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.383599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":582,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.344225+0000","last_change":"2024-03-04T16:15:01.756693+0000","last_active":"2024-03-04T16:17:42.344225+0000","last_peered":"2024-03-04T16:17:42.344225+0000","last_clean":"2024-03-04T16:17:42.344225+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:17:42.344225+0000","last_undegraded":"2024-03-04T16:17:42.344225+0000","last_fullsized":"2024-03-04T16:17:42.344225+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:01.756601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:01.756601+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":389,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332259+0000","last_change":"2024-03-04T16:16:16.701023+0000","last_active":"2024-03-04T16:17:42.332259+0000","last_peered":"2024-03-04T16:17:42.332259+0000","last_clean":"2024-03-04T16:17:42.332259+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:17:42.332259+0000","last_undegraded":"2024-03-04T16:17:42.332259+0000","last_fullsized":"2024-03-04T16:17:42.332259+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.700854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.700854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660443+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347305+0000","last_change":"2024-03-04T16:16:05.072152+0000","last_active":"2024-03-04T16:17:42.347305+0000","last_peered":"2024-03-04T16:17:42.347305+0000","last_clean":"2024-03-04T16:17:42.347305+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:17:42.347305+0000","last_undegraded":"2024-03-04T16:17:42.347305+0000","last_fullsized":"2024-03-04T16:17:42.347305+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:05.071956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:05.071956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335637+0000","last_change":"2024-03-04T16:15:49.352442+0000","last_active":"2024-03-04T16:17:42.335637+0000","last_peered":"2024-03-04T16:17:42.335637+0000","last_clean":"2024-03-04T16:17:42.335637+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:17:42.335637+0000","last_undegraded":"2024-03-04T16:17:42.335637+0000","last_fullsized":"2024-03-04T16:17:42.335637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:49.352334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:49.352334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265112+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":610,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.340379+0000","last_change":"2024-03-04T16:15:49.244892+0000","last_active":"2024-03-04T16:17:42.340379+0000","last_peered":"2024-03-04T16:17:42.340379+0000","last_clean":"2024-03-04T16:17:42.340379+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:17:42.340379+0000","last_undegraded":"2024-03-04T16:17:42.340379+0000","last_fullsized":"2024-03-04T16:17:42.340379+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:49.244760+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:15:49.244760+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284123+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343524+0000","last_change":"2024-03-04T16:16:00.082788+0000","last_active":"2024-03-04T16:17:42.343524+0000","last_peered":"2024-03-04T16:17:42.343524+0000","last_clean":"2024-03-04T16:17:42.343524+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:17:42.343524+0000","last_undegraded":"2024-03-04T16:17:42.343524+0000","last_fullsized":"2024-03-04T16:17:42.343524+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.082581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190567+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.340182+0000","last_change":"2024-03-04T16:16:10.363294+0000","last_active":"2024-03-04T16:17:42.340182+0000","last_peered":"2024-03-04T16:17:42.340182+0000","last_clean":"2024-03-04T16:17:42.340182+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:17:42.340182+0000","last_undegraded":"2024-03-04T16:17:42.340182+0000","last_fullsized":"2024-03-04T16:17:42.340182+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.363176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.363176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326154+0000","last_change":"2024-03-04T16:16:57.350376+0000","last_active":"2024-03-04T16:17:42.326154+0000","last_peered":"2024-03-04T16:17:42.326154+0000","last_clean":"2024-03-04T16:17:42.326154+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:17:42.326154+0000","last_undegraded":"2024-03-04T16:17:42.326154+0000","last_fullsized":"2024-03-04T16:17:42.326154+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.350240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.350240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.159288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":598,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326483+0000","last_change":"2024-03-04T16:15:56.553897+0000","last_active":"2024-03-04T16:17:42.326483+0000","last_peered":"2024-03-04T16:17:42.326483+0000","last_clean":"2024-03-04T16:17:42.326483+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:17:42.326483+0000","last_undegraded":"2024-03-04T16:17:42.326483+0000","last_fullsized":"2024-03-04T16:17:42.326483+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:56.553745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.553745+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":422,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335791+0000","last_change":"2024-03-04T16:16:57.407575+0000","last_active":"2024-03-04T16:17:42.335791+0000","last_peered":"2024-03-04T16:17:42.335791+0000","last_clean":"2024-03-04T16:17:42.335791+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:17:42.335791+0000","last_undegraded":"2024-03-04T16:17:42.335791+0000","last_fullsized":"2024-03-04T16:17:42.335791+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.407499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.407499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.214984+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":754,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336805+0000","last_change":"2024-03-04T16:16:59.613637+0000","last_active":"2024-03-04T16:17:42.336805+0000","last_peered":"2024-03-04T16:17:42.336805+0000","last_clean":"2024-03-04T16:17:42.336805+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:17:42.336805+0000","last_undegraded":"2024-03-04T16:17:42.336805+0000","last_fullsized":"2024-03-04T16:17:42.336805+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:16:59.613549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.613549+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:08.905918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":587,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335058+0000","last_change":"2024-03-04T16:16:14.389925+0000","last_active":"2024-03-04T16:17:42.335058+0000","last_peered":"2024-03-04T16:17:42.335058+0000","last_clean":"2024-03-04T16:17:42.335058+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:17:42.335058+0000","last_undegraded":"2024-03-04T16:17:42.335058+0000","last_fullsized":"2024-03-04T16:17:42.335058+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:14.389807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.389807+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.221837+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":599,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.349300+0000","last_change":"2024-03-04T16:16:53.775738+0000","last_active":"2024-03-04T16:17:42.349300+0000","last_peered":"2024-03-04T16:17:42.349300+0000","last_clean":"2024-03-04T16:17:42.349300+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:17:42.349300+0000","last_undegraded":"2024-03-04T16:17:42.349300+0000","last_fullsized":"2024-03-04T16:17:42.349300+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:53.775609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.775609+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.927920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.327060+0000","last_change":"2024-03-04T16:15:10.975909+0000","last_active":"2024-03-04T16:17:42.327060+0000","last_peered":"2024-03-04T16:17:42.327060+0000","last_clean":"2024-03-04T16:17:42.327060+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:17:42.327060+0000","last_undegraded":"2024-03-04T16:17:42.327060+0000","last_fullsized":"2024-03-04T16:17:42.327060+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:10.975757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:10.975757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":728,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.350219+0000","last_change":"2024-03-04T16:15:10.811100+0000","last_active":"2024-03-04T16:17:42.350219+0000","last_peered":"2024-03-04T16:17:42.350219+0000","last_clean":"2024-03-04T16:17:42.350219+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:17:42.350219+0000","last_undegraded":"2024-03-04T16:17:42.350219+0000","last_fullsized":"2024-03-04T16:17:42.350219+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:15:10.810975+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:15:10.810975+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190485+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":446,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332690+0000","last_change":"2024-03-04T16:16:04.396775+0000","last_active":"2024-03-04T16:17:42.332690+0000","last_peered":"2024-03-04T16:17:42.332690+0000","last_clean":"2024-03-04T16:17:42.332690+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:17:42.332690+0000","last_undegraded":"2024-03-04T16:17:42.332690+0000","last_fullsized":"2024-03-04T16:17:42.332690+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:04.396254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:04.396254+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265078+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":573,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337038+0000","last_change":"2024-03-04T16:15:07.770440+0000","last_active":"2024-03-04T16:17:42.337038+0000","last_peered":"2024-03-04T16:17:42.337038+0000","last_clean":"2024-03-04T16:17:42.337038+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:17:42.337038+0000","last_undegraded":"2024-03-04T16:17:42.337038+0000","last_fullsized":"2024-03-04T16:17:42.337038+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:15:07.770201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:07.770201+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.453958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":265,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326371+0000","last_change":"2024-03-04T16:14:50.760982+0000","last_active":"2024-03-04T16:17:42.326371+0000","last_peered":"2024-03-04T16:17:42.326371+0000","last_clean":"2024-03-04T16:17:42.326371+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:17:42.326371+0000","last_undegraded":"2024-03-04T16:17:42.326371+0000","last_fullsized":"2024-03-04T16:17:42.326371+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:50.760883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:50.760883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":529,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.346974+0000","last_change":"2024-03-04T16:14:49.729462+0000","last_active":"2024-03-04T16:17:42.346974+0000","last_peered":"2024-03-04T16:17:42.346974+0000","last_clean":"2024-03-04T16:17:42.346974+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:17:42.346974+0000","last_undegraded":"2024-03-04T16:17:42.346974+0000","last_fullsized":"2024-03-04T16:17:42.346974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:49.729334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:49.729334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190300+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.333084+0000","last_change":"2024-03-04T16:16:55.383375+0000","last_active":"2024-03-04T16:17:42.333084+0000","last_peered":"2024-03-04T16:17:42.333084+0000","last_clean":"2024-03-04T16:17:42.333084+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:17:42.333084+0000","last_undegraded":"2024-03-04T16:17:42.333084+0000","last_fullsized":"2024-03-04T16:17:42.333084+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.383292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.383292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.382040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338275+0000","last_change":"2024-03-04T16:16:53.856464+0000","last_active":"2024-03-04T16:17:42.338275+0000","last_peered":"2024-03-04T16:17:42.338275+0000","last_clean":"2024-03-04T16:17:42.338275+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:17:42.338275+0000","last_undegraded":"2024-03-04T16:17:42.338275+0000","last_fullsized":"2024-03-04T16:17:42.338275+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:16:53.856361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.856361+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.016107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335533+0000","last_change":"2024-03-04T16:16:56.475032+0000","last_active":"2024-03-04T16:17:42.335533+0000","last_peered":"2024-03-04T16:17:42.335533+0000","last_clean":"2024-03-04T16:17:42.335533+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:17:42.335533+0000","last_undegraded":"2024-03-04T16:17:42.335533+0000","last_fullsized":"2024-03-04T16:17:42.335533+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:56.474881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:16:56.474881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.070090+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338802+0000","last_change":"2024-03-04T16:15:56.777453+0000","last_active":"2024-03-04T16:17:42.338802+0000","last_peered":"2024-03-04T16:17:42.338802+0000","last_clean":"2024-03-04T16:17:42.338802+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:17:42.338802+0000","last_undegraded":"2024-03-04T16:17:42.338802+0000","last_fullsized":"2024-03-04T16:17:42.338802+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:56.777347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.777347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454230+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":459,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332689+0000","last_change":"2024-03-04T16:16:51.831340+0000","last_active":"2024-03-04T16:17:42.332689+0000","last_peered":"2024-03-04T16:17:42.332689+0000","last_clean":"2024-03-04T16:17:42.332689+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:17:42.332689+0000","last_undegraded":"2024-03-04T16:17:42.332689+0000","last_fullsized":"2024-03-04T16:17:42.332689+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:51.830496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:16:51.830496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.535280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347171+0000","last_change":"2024-03-04T16:14:53.778826+0000","last_active":"2024-03-04T16:17:42.347171+0000","last_peered":"2024-03-04T16:17:42.347171+0000","last_clean":"2024-03-04T16:17:42.347171+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:17:42.347171+0000","last_undegraded":"2024-03-04T16:17:42.347171+0000","last_fullsized":"2024-03-04T16:17:42.347171+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:14:53.778693+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.778693+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190332+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"312'4529","reported_seq":4571,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.945160+0000","last_change":"2024-03-04T16:16:53.660438+0000","last_active":"2024-03-04T16:17:42.945160+0000","last_peered":"2024-03-04T16:17:42.945160+0000","last_clean":"2024-03-04T16:17:42.945160+0000","last_became_active":"2024-03-04T16:16:53.660156+0000","last_became_peered":"2024-03-04T16:16:53.660156+0000","last_unstale":"2024-03-04T16:17:42.945160+0000","last_undegraded":"2024-03-04T16:17:42.945160+0000","last_fullsized":"2024-03-04T16:17:42.945160+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4529,"log_dups_size":0,"ondisk_log_size":4529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.297306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296812544,"num_objects":4529,"num_object_clones":0,"num_object_copies":13587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":289856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.340246+0000","last_change":"2024-03-04T16:15:48.981473+0000","last_active":"2024-03-04T16:17:42.340246+0000","last_peered":"2024-03-04T16:17:42.340246+0000","last_clean":"2024-03-04T16:17:42.340246+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:17:42.340246+0000","last_undegraded":"2024-03-04T16:17:42.340246+0000","last_fullsized":"2024-03-04T16:17:42.340246+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:48.981357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.981357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454050+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":597,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336434+0000","last_change":"2024-03-04T16:17:07.251362+0000","last_active":"2024-03-04T16:17:42.336434+0000","last_peered":"2024-03-04T16:17:42.336434+0000","last_clean":"2024-03-04T16:17:42.336434+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:17:42.336434+0000","last_undegraded":"2024-03-04T16:17:42.336434+0000","last_fullsized":"2024-03-04T16:17:42.336434+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:17:07.251155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.251155+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.475039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":613,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336199+0000","last_change":"2024-03-04T16:14:45.845126+0000","last_active":"2024-03-04T16:17:42.336199+0000","last_peered":"2024-03-04T16:17:42.336199+0000","last_clean":"2024-03-04T16:17:42.336199+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:17:42.336199+0000","last_undegraded":"2024-03-04T16:17:42.336199+0000","last_fullsized":"2024-03-04T16:17:42.336199+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:14:45.844973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:14:45.844973+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.453927+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":751,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.334882+0000","last_change":"2024-03-04T16:17:09.268256+0000","last_active":"2024-03-04T16:17:42.334882+0000","last_peered":"2024-03-04T16:17:42.334882+0000","last_clean":"2024-03-04T16:17:42.334882+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:17:42.334882+0000","last_undegraded":"2024-03-04T16:17:42.334882+0000","last_fullsized":"2024-03-04T16:17:42.334882+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:17:09.268137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:09.268137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.542889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"312'4523","reported_seq":4565,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.864153+0000","last_change":"2024-03-04T16:16:53.663167+0000","last_active":"2024-03-04T16:17:43.864153+0000","last_peered":"2024-03-04T16:17:43.864153+0000","last_clean":"2024-03-04T16:17:43.864153+0000","last_became_active":"2024-03-04T16:16:53.661370+0000","last_became_peered":"2024-03-04T16:16:53.661370+0000","last_unstale":"2024-03-04T16:17:43.864153+0000","last_undegraded":"2024-03-04T16:17:43.864153+0000","last_fullsized":"2024-03-04T16:17:43.864153+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.342888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296419328,"num_objects":4523,"num_object_clones":0,"num_object_copies":13569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4523,"num_write_kb":289472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347468+0000","last_change":"2024-03-04T16:16:18.274771+0000","last_active":"2024-03-04T16:17:42.347468+0000","last_peered":"2024-03-04T16:17:42.347468+0000","last_clean":"2024-03-04T16:17:42.347468+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:17:42.347468+0000","last_undegraded":"2024-03-04T16:17:42.347468+0000","last_fullsized":"2024-03-04T16:17:42.347468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.274659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.274659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190740+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":561,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335984+0000","last_change":"2024-03-04T16:16:20.383892+0000","last_active":"2024-03-04T16:17:42.335984+0000","last_peered":"2024-03-04T16:17:42.335984+0000","last_clean":"2024-03-04T16:17:42.335984+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:17:42.335984+0000","last_undegraded":"2024-03-04T16:17:42.335984+0000","last_fullsized":"2024-03-04T16:17:42.335984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.383782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.383782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.706926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":581,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326699+0000","last_change":"2024-03-04T16:16:08.552789+0000","last_active":"2024-03-04T16:17:42.326699+0000","last_peered":"2024-03-04T16:17:42.326699+0000","last_clean":"2024-03-04T16:17:42.326699+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:17:42.326699+0000","last_undegraded":"2024-03-04T16:17:42.326699+0000","last_fullsized":"2024-03-04T16:17:42.326699+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:16:08.552635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.552635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":53,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343229+0000","last_change":"2024-03-04T16:16:24.044051+0000","last_active":"2024-03-04T16:17:42.343229+0000","last_peered":"2024-03-04T16:17:42.343229+0000","last_clean":"2024-03-04T16:17:42.343229+0000","last_became_active":"2024-03-04T16:16:24.043673+0000","last_became_peered":"2024-03-04T16:16:24.043673+0000","last_unstale":"2024-03-04T16:17:42.343229+0000","last_undegraded":"2024-03-04T16:17:42.343229+0000","last_fullsized":"2024-03-04T16:17:42.343229+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:46.262241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:46.262241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190539+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"312'4430","reported_seq":4472,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.515755+0000","last_change":"2024-03-04T16:16:53.660736+0000","last_active":"2024-03-04T16:17:43.515755+0000","last_peered":"2024-03-04T16:17:43.515755+0000","last_clean":"2024-03-04T16:17:43.515755+0000","last_became_active":"2024-03-04T16:16:53.660464+0000","last_became_peered":"2024-03-04T16:16:53.660464+0000","last_unstale":"2024-03-04T16:17:43.515755+0000","last_undegraded":"2024-03-04T16:17:43.515755+0000","last_fullsized":"2024-03-04T16:17:43.515755+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:01.967586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290324480,"num_objects":4430,"num_object_clones":0,"num_object_copies":13290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4430,"num_write_kb":283520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332763+0000","last_change":"2024-03-04T16:16:18.374475+0000","last_active":"2024-03-04T16:17:42.332763+0000","last_peered":"2024-03-04T16:17:42.332763+0000","last_clean":"2024-03-04T16:17:42.332763+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:17:42.332763+0000","last_undegraded":"2024-03-04T16:17:42.332763+0000","last_fullsized":"2024-03-04T16:17:42.332763+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.374381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.374381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265127+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":281,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336862+0000","last_change":"2024-03-04T16:16:19.642828+0000","last_active":"2024-03-04T16:17:42.336862+0000","last_peered":"2024-03-04T16:17:42.336862+0000","last_clean":"2024-03-04T16:17:42.336862+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:17:42.336862+0000","last_undegraded":"2024-03-04T16:17:42.336862+0000","last_fullsized":"2024-03-04T16:17:42.336862+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.642722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.642722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660508+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335896+0000","last_change":"2024-03-04T16:15:46.375861+0000","last_active":"2024-03-04T16:17:42.335896+0000","last_peered":"2024-03-04T16:17:42.335896+0000","last_clean":"2024-03-04T16:17:42.335896+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:17:42.335896+0000","last_undegraded":"2024-03-04T16:17:42.335896+0000","last_fullsized":"2024-03-04T16:17:42.335896+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:46.375761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:15:46.375761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":360,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.340500+0000","last_change":"2024-03-04T16:16:00.381546+0000","last_active":"2024-03-04T16:17:42.340500+0000","last_peered":"2024-03-04T16:17:42.340500+0000","last_clean":"2024-03-04T16:17:42.340500+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:17:42.340500+0000","last_undegraded":"2024-03-04T16:17:42.340500+0000","last_fullsized":"2024-03-04T16:17:42.340500+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:16:00.381394+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.381394+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284210+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"312'4528","reported_seq":4570,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.507772+0000","last_change":"2024-03-04T16:16:53.660460+0000","last_active":"2024-03-04T16:17:43.507772+0000","last_peered":"2024-03-04T16:17:43.507772+0000","last_clean":"2024-03-04T16:17:43.507772+0000","last_became_active":"2024-03-04T16:16:53.660170+0000","last_became_peered":"2024-03-04T16:16:53.660170+0000","last_unstale":"2024-03-04T16:17:43.507772+0000","last_undegraded":"2024-03-04T16:17:43.507772+0000","last_fullsized":"2024-03-04T16:17:43.507772+0000","mapping_epoch":296,"log_start":"302'600","ondisk_log_start":"302'600","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3928,"log_dups_size":0,"ondisk_log_size":3928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.653120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296747008,"num_objects":4528,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":289792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":428,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332561+0000","last_change":"2024-03-04T16:16:12.401505+0000","last_active":"2024-03-04T16:17:42.332561+0000","last_peered":"2024-03-04T16:17:42.332561+0000","last_clean":"2024-03-04T16:17:42.332561+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:17:42.332561+0000","last_undegraded":"2024-03-04T16:17:42.332561+0000","last_fullsized":"2024-03-04T16:17:42.332561+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.401385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.221777+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":265,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326352+0000","last_change":"2024-03-04T16:14:47.655351+0000","last_active":"2024-03-04T16:17:42.326352+0000","last_peered":"2024-03-04T16:17:42.326352+0000","last_clean":"2024-03-04T16:17:42.326352+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:17:42.326352+0000","last_undegraded":"2024-03-04T16:17:42.326352+0000","last_fullsized":"2024-03-04T16:17:42.326352+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:47.655216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:14:47.655216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":558,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337087+0000","last_change":"2024-03-04T16:15:53.841693+0000","last_active":"2024-03-04T16:17:42.337087+0000","last_peered":"2024-03-04T16:17:42.337087+0000","last_clean":"2024-03-04T16:17:42.337087+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:17:42.337087+0000","last_undegraded":"2024-03-04T16:17:42.337087+0000","last_fullsized":"2024-03-04T16:17:42.337087+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:53.841578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:15:53.841578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454020+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336710+0000","last_change":"2024-03-04T16:16:49.762334+0000","last_active":"2024-03-04T16:17:42.336710+0000","last_peered":"2024-03-04T16:17:42.336710+0000","last_clean":"2024-03-04T16:17:42.336710+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:17:42.336710+0000","last_undegraded":"2024-03-04T16:17:42.336710+0000","last_fullsized":"2024-03-04T16:17:42.336710+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:16:49.762239+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.762239+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.560260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"312'4442","reported_seq":4484,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.506617+0000","last_change":"2024-03-04T16:16:53.660429+0000","last_active":"2024-03-04T16:17:43.506617+0000","last_peered":"2024-03-04T16:17:43.506617+0000","last_clean":"2024-03-04T16:17:43.506617+0000","last_became_active":"2024-03-04T16:16:53.660115+0000","last_became_peered":"2024-03-04T16:16:53.660115+0000","last_unstale":"2024-03-04T16:17:43.506617+0000","last_undegraded":"2024-03-04T16:17:43.506617+0000","last_fullsized":"2024-03-04T16:17:43.506617+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3942,"log_dups_size":0,"ondisk_log_size":3942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.638211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291110912,"num_objects":4442,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":284288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":525,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.346803+0000","last_change":"2024-03-04T16:14:51.811044+0000","last_active":"2024-03-04T16:17:42.346803+0000","last_peered":"2024-03-04T16:17:42.346803+0000","last_clean":"2024-03-04T16:17:42.346803+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:17:42.346803+0000","last_undegraded":"2024-03-04T16:17:42.346803+0000","last_fullsized":"2024-03-04T16:17:42.346803+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:51.810913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.810913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190360+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":574,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339788+0000","last_change":"2024-03-04T16:15:51.894739+0000","last_active":"2024-03-04T16:17:42.339788+0000","last_peered":"2024-03-04T16:17:42.339788+0000","last_clean":"2024-03-04T16:17:42.339788+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:17:42.339788+0000","last_undegraded":"2024-03-04T16:17:42.339788+0000","last_fullsized":"2024-03-04T16:17:42.339788+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:15:51.894452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.894452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454084+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":572,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326133+0000","last_change":"2024-03-04T16:15:45.509761+0000","last_active":"2024-03-04T16:17:42.326133+0000","last_peered":"2024-03-04T16:17:42.326133+0000","last_clean":"2024-03-04T16:17:42.326133+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:17:42.326133+0000","last_undegraded":"2024-03-04T16:17:42.326133+0000","last_fullsized":"2024-03-04T16:17:42.326133+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:45.509639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:45.509639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":852,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332492+0000","last_change":"2024-03-04T16:16:17.407977+0000","last_active":"2024-03-04T16:17:42.332492+0000","last_peered":"2024-03-04T16:17:42.332492+0000","last_clean":"2024-03-04T16:17:42.332492+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:17:42.332492+0000","last_undegraded":"2024-03-04T16:17:42.332492+0000","last_fullsized":"2024-03-04T16:17:42.332492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:16:17.407775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.407775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284324+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":397,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338939+0000","last_change":"2024-03-04T16:17:01.832332+0000","last_active":"2024-03-04T16:17:42.338939+0000","last_peered":"2024-03-04T16:17:42.338939+0000","last_clean":"2024-03-04T16:17:42.338939+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:17:42.338939+0000","last_undegraded":"2024-03-04T16:17:42.338939+0000","last_fullsized":"2024-03-04T16:17:42.338939+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.832115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.832115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:27.727695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"312'4551","reported_seq":4593,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.969696+0000","last_change":"2024-03-04T16:16:53.657683+0000","last_active":"2024-03-04T16:17:42.969696+0000","last_peered":"2024-03-04T16:17:42.969696+0000","last_clean":"2024-03-04T16:17:42.969696+0000","last_became_active":"2024-03-04T16:16:53.657368+0000","last_became_peered":"2024-03-04T16:16:53.657368+0000","last_unstale":"2024-03-04T16:17:42.969696+0000","last_undegraded":"2024-03-04T16:17:42.969696+0000","last_fullsized":"2024-03-04T16:17:42.969696+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:55.076373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298254336,"num_objects":4551,"num_object_clones":0,"num_object_copies":13653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4551,"num_write_kb":291264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":548,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336047+0000","last_change":"2024-03-04T16:16:02.471635+0000","last_active":"2024-03-04T16:17:42.336047+0000","last_peered":"2024-03-04T16:17:42.336047+0000","last_clean":"2024-03-04T16:17:42.336047+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:17:42.336047+0000","last_undegraded":"2024-03-04T16:17:42.336047+0000","last_fullsized":"2024-03-04T16:17:42.336047+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.471522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.471522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265097+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.334636+0000","last_change":"2024-03-04T16:16:14.435854+0000","last_active":"2024-03-04T16:17:42.334636+0000","last_peered":"2024-03-04T16:17:42.334636+0000","last_clean":"2024-03-04T16:17:42.334636+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:17:42.334636+0000","last_undegraded":"2024-03-04T16:17:42.334636+0000","last_fullsized":"2024-03-04T16:17:42.334636+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:14.435737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.435737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":274,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343625+0000","last_change":"2024-03-04T16:16:10.021523+0000","last_active":"2024-03-04T16:17:42.343625+0000","last_peered":"2024-03-04T16:17:42.343625+0000","last_clean":"2024-03-04T16:17:42.343625+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:17:42.343625+0000","last_undegraded":"2024-03-04T16:17:42.343625+0000","last_fullsized":"2024-03-04T16:17:42.343625+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.021439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190768+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":197,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332069+0000","last_change":"2024-03-04T16:16:57.389891+0000","last_active":"2024-03-04T16:17:42.332069+0000","last_peered":"2024-03-04T16:17:42.332069+0000","last_clean":"2024-03-04T16:17:42.332069+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:17:42.332069+0000","last_undegraded":"2024-03-04T16:17:42.332069+0000","last_fullsized":"2024-03-04T16:17:42.332069+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:16:57.389758+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.389758+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.714260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":273,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339065+0000","last_change":"2024-03-04T16:16:19.696457+0000","last_active":"2024-03-04T16:17:42.339065+0000","last_peered":"2024-03-04T16:17:42.339065+0000","last_clean":"2024-03-04T16:17:42.339065+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:17:42.339065+0000","last_undegraded":"2024-03-04T16:17:42.339065+0000","last_fullsized":"2024-03-04T16:17:42.339065+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.696369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.696369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454331+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"312'4522","reported_seq":4563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.696074+0000","last_change":"2024-03-04T16:16:53.666174+0000","last_active":"2024-03-04T16:17:43.696074+0000","last_peered":"2024-03-04T16:17:43.696074+0000","last_clean":"2024-03-04T16:17:43.696074+0000","last_became_active":"2024-03-04T16:16:53.665651+0000","last_became_peered":"2024-03-04T16:16:53.665651+0000","last_unstale":"2024-03-04T16:17:43.696074+0000","last_undegraded":"2024-03-04T16:17:43.696074+0000","last_fullsized":"2024-03-04T16:17:43.696074+0000","mapping_epoch":296,"log_start":"302'700","ondisk_log_start":"302'700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.126117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296353792,"num_objects":4522,"num_object_clones":0,"num_object_copies":13566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4522,"num_write_kb":289408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339440+0000","last_change":"2024-03-04T16:14:57.859262+0000","last_active":"2024-03-04T16:17:42.339440+0000","last_peered":"2024-03-04T16:17:42.339440+0000","last_clean":"2024-03-04T16:17:42.339440+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:17:42.339440+0000","last_undegraded":"2024-03-04T16:17:42.339440+0000","last_fullsized":"2024-03-04T16:17:42.339440+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:57.859146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:57.859146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.453865+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":861,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337407+0000","last_change":"2024-03-04T16:16:01.374602+0000","last_active":"2024-03-04T16:17:42.337407+0000","last_peered":"2024-03-04T16:17:42.337407+0000","last_clean":"2024-03-04T16:17:42.337407+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:17:42.337407+0000","last_undegraded":"2024-03-04T16:17:42.337407+0000","last_fullsized":"2024-03-04T16:17:42.337407+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:16:01.374477+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.374477+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265178+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343578+0000","last_change":"2024-03-04T16:16:20.104755+0000","last_active":"2024-03-04T16:17:42.343578+0000","last_peered":"2024-03-04T16:17:42.343578+0000","last_clean":"2024-03-04T16:17:42.343578+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:17:42.343578+0000","last_undegraded":"2024-03-04T16:17:42.343578+0000","last_fullsized":"2024-03-04T16:17:42.343578+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.104643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.104643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":431,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326650+0000","last_change":"2024-03-04T16:17:05.303664+0000","last_active":"2024-03-04T16:17:42.326650+0000","last_peered":"2024-03-04T16:17:42.326650+0000","last_clean":"2024-03-04T16:17:42.326650+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:17:42.326650+0000","last_undegraded":"2024-03-04T16:17:42.326650+0000","last_fullsized":"2024-03-04T16:17:42.326650+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:17:05.303531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.303531+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.512468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":560,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336891+0000","last_change":"2024-03-04T16:16:15.349131+0000","last_active":"2024-03-04T16:17:42.336891+0000","last_peered":"2024-03-04T16:17:42.336891+0000","last_clean":"2024-03-04T16:17:42.336891+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:17:42.336891+0000","last_undegraded":"2024-03-04T16:17:42.336891+0000","last_fullsized":"2024-03-04T16:17:42.336891+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:15.348942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.348942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"312'4597","reported_seq":4639,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.647782+0000","last_change":"2024-03-04T16:16:53.660223+0000","last_active":"2024-03-04T16:17:42.647782+0000","last_peered":"2024-03-04T16:17:42.647782+0000","last_clean":"2024-03-04T16:17:42.647782+0000","last_became_active":"2024-03-04T16:16:53.659965+0000","last_became_peered":"2024-03-04T16:16:53.659965+0000","last_unstale":"2024-03-04T16:17:42.647782+0000","last_undegraded":"2024-03-04T16:17:42.647782+0000","last_fullsized":"2024-03-04T16:17:42.647782+0000","mapping_epoch":296,"log_start":"301'300","ondisk_log_start":"301'300","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4297,"log_dups_size":0,"ondisk_log_size":4297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.464393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301268992,"num_objects":4597,"num_object_clones":0,"num_object_copies":13791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4597,"num_write_kb":294208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332083+0000","last_change":"2024-03-04T16:16:55.443627+0000","last_active":"2024-03-04T16:17:42.332083+0000","last_peered":"2024-03-04T16:17:42.332083+0000","last_clean":"2024-03-04T16:17:42.332083+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:17:42.332083+0000","last_undegraded":"2024-03-04T16:17:42.332083+0000","last_fullsized":"2024-03-04T16:17:42.332083+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.443547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.443547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.590784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":405,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339245+0000","last_change":"2024-03-04T16:17:02.856208+0000","last_active":"2024-03-04T16:17:42.339245+0000","last_peered":"2024-03-04T16:17:42.339245+0000","last_clean":"2024-03-04T16:17:42.339245+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:17:42.339245+0000","last_undegraded":"2024-03-04T16:17:42.339245+0000","last_fullsized":"2024-03-04T16:17:42.339245+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:02.856124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:02.856124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:04.756539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1467,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.337141+0000","last_change":"2024-03-04T16:16:10.322558+0000","last_active":"2024-03-04T16:17:42.337141+0000","last_peered":"2024-03-04T16:17:42.337141+0000","last_clean":"2024-03-04T16:17:42.337141+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:17:42.337141+0000","last_undegraded":"2024-03-04T16:17:42.337141+0000","last_fullsized":"2024-03-04T16:17:42.337141+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:16:10.322412+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.322412+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265146+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":892,"num_read_kb":887,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":441,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338008+0000","last_change":"2024-03-04T16:15:43.291569+0000","last_active":"2024-03-04T16:17:42.338008+0000","last_peered":"2024-03-04T16:17:42.338008+0000","last_clean":"2024-03-04T16:17:42.338008+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:17:42.338008+0000","last_undegraded":"2024-03-04T16:17:42.338008+0000","last_fullsized":"2024-03-04T16:17:42.338008+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:15:43.291436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:43.291436+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660382+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339851+0000","last_change":"2024-03-04T16:16:13.854634+0000","last_active":"2024-03-04T16:17:42.339851+0000","last_peered":"2024-03-04T16:17:42.339851+0000","last_clean":"2024-03-04T16:17:42.339851+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:17:42.339851+0000","last_undegraded":"2024-03-04T16:17:42.339851+0000","last_fullsized":"2024-03-04T16:17:42.339851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:13.854523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:13.854523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:42.457442+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"312'4403","reported_seq":4445,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.702944+0000","last_change":"2024-03-04T16:16:53.661446+0000","last_active":"2024-03-04T16:17:42.702944+0000","last_peered":"2024-03-04T16:17:42.702944+0000","last_clean":"2024-03-04T16:17:42.702944+0000","last_became_active":"2024-03-04T16:16:53.661113+0000","last_became_peered":"2024-03-04T16:16:53.661113+0000","last_unstale":"2024-03-04T16:17:42.702944+0000","last_undegraded":"2024-03-04T16:17:42.702944+0000","last_fullsized":"2024-03-04T16:17:42.702944+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.556881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":281792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.331618+0000","last_change":"2024-03-04T16:16:11.401122+0000","last_active":"2024-03-04T16:17:42.331618+0000","last_peered":"2024-03-04T16:17:42.331618+0000","last_clean":"2024-03-04T16:17:42.331618+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:17:42.331618+0000","last_undegraded":"2024-03-04T16:17:42.331618+0000","last_fullsized":"2024-03-04T16:17:42.331618+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.401004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.401004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284262+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332343+0000","last_change":"2024-03-04T16:16:04.339942+0000","last_active":"2024-03-04T16:17:42.332343+0000","last_peered":"2024-03-04T16:17:42.332343+0000","last_clean":"2024-03-04T16:17:42.332343+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:17:42.332343+0000","last_undegraded":"2024-03-04T16:17:42.332343+0000","last_fullsized":"2024-03-04T16:17:42.332343+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:04.339827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:04.339827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284291+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":528,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335955+0000","last_change":"2024-03-04T16:14:53.705612+0000","last_active":"2024-03-04T16:17:42.335955+0000","last_peered":"2024-03-04T16:17:42.335955+0000","last_clean":"2024-03-04T16:17:42.335955+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:17:42.335955+0000","last_undegraded":"2024-03-04T16:17:42.335955+0000","last_fullsized":"2024-03-04T16:17:42.335955+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:14:53.705474+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.705474+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.453991+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":265,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339879+0000","last_change":"2024-03-04T16:16:54.600353+0000","last_active":"2024-03-04T16:17:42.339879+0000","last_peered":"2024-03-04T16:17:42.339879+0000","last_clean":"2024-03-04T16:17:42.339879+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:17:42.339879+0000","last_undegraded":"2024-03-04T16:17:42.339879+0000","last_fullsized":"2024-03-04T16:17:42.339879+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:54.600256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:54.600256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.964184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"312'4383","reported_seq":4424,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.689050+0000","last_change":"2024-03-04T16:16:53.660377+0000","last_active":"2024-03-04T16:17:42.689050+0000","last_peered":"2024-03-04T16:17:42.689050+0000","last_clean":"2024-03-04T16:17:42.689050+0000","last_became_active":"2024-03-04T16:16:53.659725+0000","last_became_peered":"2024-03-04T16:16:53.659725+0000","last_unstale":"2024-03-04T16:17:42.689050+0000","last_undegraded":"2024-03-04T16:17:42.689050+0000","last_fullsized":"2024-03-04T16:17:42.689050+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.365457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287244288,"num_objects":4383,"num_object_clones":0,"num_object_copies":13149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":280512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":556,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332365+0000","last_change":"2024-03-04T16:15:52.388804+0000","last_active":"2024-03-04T16:17:42.332365+0000","last_peered":"2024-03-04T16:17:42.332365+0000","last_clean":"2024-03-04T16:17:42.332365+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:17:42.332365+0000","last_undegraded":"2024-03-04T16:17:42.332365+0000","last_fullsized":"2024-03-04T16:17:42.332365+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:52.388669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:52.388669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660428+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":565,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336903+0000","last_change":"2024-03-04T16:16:01.874625+0000","last_active":"2024-03-04T16:17:42.336903+0000","last_peered":"2024-03-04T16:17:42.336903+0000","last_clean":"2024-03-04T16:17:42.336903+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:17:42.336903+0000","last_undegraded":"2024-03-04T16:17:42.336903+0000","last_fullsized":"2024-03-04T16:17:42.336903+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:01.874356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.874356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":585,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.340277+0000","last_change":"2024-03-04T16:15:47.218861+0000","last_active":"2024-03-04T16:17:42.340277+0000","last_peered":"2024-03-04T16:17:42.340277+0000","last_clean":"2024-03-04T16:17:42.340277+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:17:42.340277+0000","last_undegraded":"2024-03-04T16:17:42.340277+0000","last_fullsized":"2024-03-04T16:17:42.340277+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:15:47.218155+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:15:47.218155+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284180+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":381,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336176+0000","last_change":"2024-03-04T16:16:11.811999+0000","last_active":"2024-03-04T16:17:42.336176+0000","last_peered":"2024-03-04T16:17:42.336176+0000","last_clean":"2024-03-04T16:17:42.336176+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:17:42.336176+0000","last_undegraded":"2024-03-04T16:17:42.336176+0000","last_fullsized":"2024-03-04T16:17:42.336176+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:16:11.811878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.811878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"312'4543","reported_seq":4585,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.865591+0000","last_change":"2024-03-04T16:16:53.663518+0000","last_active":"2024-03-04T16:17:43.865591+0000","last_peered":"2024-03-04T16:17:43.865591+0000","last_clean":"2024-03-04T16:17:43.865591+0000","last_became_active":"2024-03-04T16:16:53.662075+0000","last_became_peered":"2024-03-04T16:16:53.662075+0000","last_unstale":"2024-03-04T16:17:43.865591+0000","last_undegraded":"2024-03-04T16:17:43.865591+0000","last_fullsized":"2024-03-04T16:17:43.865591+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.663744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297730048,"num_objects":4543,"num_object_clones":0,"num_object_copies":13629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4543,"num_write_kb":290752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":562,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335381+0000","last_change":"2024-03-04T16:16:50.462234+0000","last_active":"2024-03-04T16:17:42.335381+0000","last_peered":"2024-03-04T16:17:42.335381+0000","last_clean":"2024-03-04T16:17:42.335381+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:17:42.335381+0000","last_undegraded":"2024-03-04T16:17:42.335381+0000","last_fullsized":"2024-03-04T16:17:42.335381+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:50.462116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:16:50.462116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.011998+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":401,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.338098+0000","last_change":"2024-03-04T16:16:07.544810+0000","last_active":"2024-03-04T16:17:42.338098+0000","last_peered":"2024-03-04T16:17:42.338098+0000","last_clean":"2024-03-04T16:17:42.338098+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:17:42.338098+0000","last_undegraded":"2024-03-04T16:17:42.338098+0000","last_fullsized":"2024-03-04T16:17:42.338098+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:07.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.544679+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:46.660622+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":613,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326361+0000","last_change":"2024-03-04T16:15:48.541576+0000","last_active":"2024-03-04T16:17:42.326361+0000","last_peered":"2024-03-04T16:17:42.326361+0000","last_clean":"2024-03-04T16:17:42.326361+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:17:42.326361+0000","last_undegraded":"2024-03-04T16:17:42.326361+0000","last_fullsized":"2024-03-04T16:17:42.326361+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'35","last_scrub_stamp":"2024-03-04T16:15:48.541142+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.541142+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":726,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343941+0000","last_change":"2024-03-04T16:17:06.654928+0000","last_active":"2024-03-04T16:17:42.343941+0000","last_peered":"2024-03-04T16:17:42.343941+0000","last_clean":"2024-03-04T16:17:42.343941+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:17:42.343941+0000","last_undegraded":"2024-03-04T16:17:42.343941+0000","last_fullsized":"2024-03-04T16:17:42.343941+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:06.654784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.654784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.453080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"312'4482","reported_seq":4524,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.695686+0000","last_change":"2024-03-04T16:16:53.659484+0000","last_active":"2024-03-04T16:17:42.695686+0000","last_peered":"2024-03-04T16:17:42.695686+0000","last_clean":"2024-03-04T16:17:42.695686+0000","last_became_active":"2024-03-04T16:16:53.659122+0000","last_became_peered":"2024-03-04T16:16:53.659122+0000","last_unstale":"2024-03-04T16:17:42.695686+0000","last_undegraded":"2024-03-04T16:17:42.695686+0000","last_fullsized":"2024-03-04T16:17:42.695686+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.313098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.325955+0000","last_change":"2024-03-04T16:16:11.572661+0000","last_active":"2024-03-04T16:17:42.325955+0000","last_peered":"2024-03-04T16:17:42.325955+0000","last_clean":"2024-03-04T16:17:42.325955+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:17:42.325955+0000","last_undegraded":"2024-03-04T16:17:42.325955+0000","last_fullsized":"2024-03-04T16:17:42.325955+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.572528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.572528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":549,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.346754+0000","last_change":"2024-03-04T16:14:52.759799+0000","last_active":"2024-03-04T16:17:42.346754+0000","last_peered":"2024-03-04T16:17:42.346754+0000","last_clean":"2024-03-04T16:17:42.346754+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:17:42.346754+0000","last_undegraded":"2024-03-04T16:17:42.346754+0000","last_fullsized":"2024-03-04T16:17:42.346754+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:52.759532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:14:52.759532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":640,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343852+0000","last_change":"2024-03-04T16:15:03.777164+0000","last_active":"2024-03-04T16:17:42.343852+0000","last_peered":"2024-03-04T16:17:42.343852+0000","last_clean":"2024-03-04T16:17:42.343852+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:17:42.343852+0000","last_undegraded":"2024-03-04T16:17:42.343852+0000","last_fullsized":"2024-03-04T16:17:42.343852+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-03-04T16:15:03.776866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:03.776866+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190426+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.325945+0000","last_change":"2024-03-04T16:14:42.615690+0000","last_active":"2024-03-04T16:17:42.325945+0000","last_peered":"2024-03-04T16:17:42.325945+0000","last_clean":"2024-03-04T16:17:42.325945+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:17:42.325945+0000","last_undegraded":"2024-03-04T16:17:42.325945+0000","last_fullsized":"2024-03-04T16:17:42.325945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:42.615581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:42.615581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"312'4415","reported_seq":4457,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.695766+0000","last_change":"2024-03-04T16:16:53.660602+0000","last_active":"2024-03-04T16:17:42.695766+0000","last_peered":"2024-03-04T16:17:42.695766+0000","last_clean":"2024-03-04T16:17:42.695766+0000","last_became_active":"2024-03-04T16:16:53.660328+0000","last_became_peered":"2024-03-04T16:16:53.660328+0000","last_unstale":"2024-03-04T16:17:42.695766+0000","last_undegraded":"2024-03-04T16:17:42.695766+0000","last_fullsized":"2024-03-04T16:17:42.695766+0000","mapping_epoch":296,"log_start":"302'500","ondisk_log_start":"302'500","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3915,"log_dups_size":0,"ondisk_log_size":3915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.850706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289341440,"num_objects":4415,"num_object_clones":0,"num_object_copies":13245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4415,"num_write_kb":282560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.336142+0000","last_change":"2024-03-04T16:14:41.961081+0000","last_active":"2024-03-04T16:17:42.336142+0000","last_peered":"2024-03-04T16:17:42.336142+0000","last_clean":"2024-03-04T16:17:42.336142+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:17:42.336142+0000","last_undegraded":"2024-03-04T16:17:42.336142+0000","last_fullsized":"2024-03-04T16:17:42.336142+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:41.960969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:41.960969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.453896+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":725,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.327259+0000","last_change":"2024-03-04T16:14:59.729903+0000","last_active":"2024-03-04T16:17:42.327259+0000","last_peered":"2024-03-04T16:17:42.327259+0000","last_clean":"2024-03-04T16:17:42.327259+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:17:42.327259+0000","last_undegraded":"2024-03-04T16:17:42.327259+0000","last_fullsized":"2024-03-04T16:17:42.327259+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:14:59.729380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.729380+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.343770+0000","last_change":"2024-03-04T16:16:15.065620+0000","last_active":"2024-03-04T16:17:42.343770+0000","last_peered":"2024-03-04T16:17:42.343770+0000","last_clean":"2024-03-04T16:17:42.343770+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:17:42.343770+0000","last_undegraded":"2024-03-04T16:17:42.343770+0000","last_fullsized":"2024-03-04T16:17:42.343770+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:15.065428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.065428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"301'34","reported_seq":599,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332482+0000","last_change":"2024-03-04T16:17:05.452594+0000","last_active":"2024-03-04T16:17:42.332482+0000","last_peered":"2024-03-04T16:17:42.332482+0000","last_clean":"2024-03-04T16:17:42.332482+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:17:42.332482+0000","last_undegraded":"2024-03-04T16:17:42.332482+0000","last_fullsized":"2024-03-04T16:17:42.332482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'34","last_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.452417+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.844380+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"312'4508","reported_seq":4550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.692265+0000","last_change":"2024-03-04T16:16:53.658520+0000","last_active":"2024-03-04T16:17:43.692265+0000","last_peered":"2024-03-04T16:17:43.692265+0000","last_clean":"2024-03-04T16:17:43.692265+0000","last_became_active":"2024-03-04T16:16:53.658240+0000","last_became_peered":"2024-03-04T16:16:53.658240+0000","last_unstale":"2024-03-04T16:17:43.692265+0000","last_undegraded":"2024-03-04T16:17:43.692265+0000","last_fullsized":"2024-03-04T16:17:43.692265+0000","mapping_epoch":296,"log_start":"302'700","ondisk_log_start":"302'700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:09.724216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295436288,"num_objects":4508,"num_object_clones":0,"num_object_copies":13524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4508,"num_write_kb":288512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.349193+0000","last_change":"2024-03-04T16:15:41.965920+0000","last_active":"2024-03-04T16:17:42.349193+0000","last_peered":"2024-03-04T16:17:42.349193+0000","last_clean":"2024-03-04T16:17:42.349193+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:17:42.349193+0000","last_undegraded":"2024-03-04T16:17:42.349193+0000","last_fullsized":"2024-03-04T16:17:42.349193+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:15:41.965768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:41.965768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.339765+0000","last_change":"2024-03-04T16:15:58.806390+0000","last_active":"2024-03-04T16:17:42.339765+0000","last_peered":"2024-03-04T16:17:42.339765+0000","last_clean":"2024-03-04T16:17:42.339765+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:17:42.339765+0000","last_undegraded":"2024-03-04T16:17:42.339765+0000","last_fullsized":"2024-03-04T16:17:42.339765+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.806254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.806254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:44.454200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326156+0000","last_change":"2024-03-04T16:16:12.534941+0000","last_active":"2024-03-04T16:17:42.326156+0000","last_peered":"2024-03-04T16:17:42.326156+0000","last_clean":"2024-03-04T16:17:42.326156+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:17:42.326156+0000","last_undegraded":"2024-03-04T16:17:42.326156+0000","last_fullsized":"2024-03-04T16:17:42.326156+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.534849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.534849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":603,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.327106+0000","last_change":"2024-03-04T16:17:00.348913+0000","last_active":"2024-03-04T16:17:42.327106+0000","last_peered":"2024-03-04T16:17:42.327106+0000","last_clean":"2024-03-04T16:17:42.327106+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:17:42.327106+0000","last_undegraded":"2024-03-04T16:17:42.327106+0000","last_fullsized":"2024-03-04T16:17:42.327106+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:00.348771+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.348771+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.680455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"312'4498","reported_seq":4540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.912453+0000","last_change":"2024-03-04T16:16:53.658266+0000","last_active":"2024-03-04T16:17:42.912453+0000","last_peered":"2024-03-04T16:17:42.912453+0000","last_clean":"2024-03-04T16:17:42.912453+0000","last_became_active":"2024-03-04T16:16:53.657934+0000","last_became_peered":"2024-03-04T16:16:53.657934+0000","last_unstale":"2024-03-04T16:17:42.912453+0000","last_undegraded":"2024-03-04T16:17:42.912453+0000","last_fullsized":"2024-03-04T16:17:42.912453+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4498,"log_dups_size":0,"ondisk_log_size":4498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.803374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294780928,"num_objects":4498,"num_object_clones":0,"num_object_copies":13494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4498,"num_write_kb":287872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":279,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.346611+0000","last_change":"2024-03-04T16:16:16.301977+0000","last_active":"2024-03-04T16:17:42.346611+0000","last_peered":"2024-03-04T16:17:42.346611+0000","last_clean":"2024-03-04T16:17:42.346611+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:17:42.346611+0000","last_undegraded":"2024-03-04T16:17:42.346611+0000","last_fullsized":"2024-03-04T16:17:42.346611+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.301767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.301767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.332584+0000","last_change":"2024-03-04T16:16:07.370204+0000","last_active":"2024-03-04T16:17:42.332584+0000","last_peered":"2024-03-04T16:17:42.332584+0000","last_clean":"2024-03-04T16:17:42.332584+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:17:42.332584+0000","last_undegraded":"2024-03-04T16:17:42.332584+0000","last_fullsized":"2024-03-04T16:17:42.332584+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.370032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.370032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:47.284236+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":265,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326077+0000","last_change":"2024-03-04T16:16:02.539110+0000","last_active":"2024-03-04T16:17:42.326077+0000","last_peered":"2024-03-04T16:17:42.326077+0000","last_clean":"2024-03-04T16:17:42.326077+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:17:42.326077+0000","last_undegraded":"2024-03-04T16:17:42.326077+0000","last_fullsized":"2024-03-04T16:17:42.326077+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.538977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.538977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":646,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326518+0000","last_change":"2024-03-04T16:17:01.305039+0000","last_active":"2024-03-04T16:17:42.326518+0000","last_peered":"2024-03-04T16:17:42.326518+0000","last_clean":"2024-03-04T16:17:42.326518+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:17:42.326518+0000","last_undegraded":"2024-03-04T16:17:42.326518+0000","last_fullsized":"2024-03-04T16:17:42.326518+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:17:01.304925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.304925+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.947920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"312'4548","reported_seq":4590,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:43.670445+0000","last_change":"2024-03-04T16:16:53.661912+0000","last_active":"2024-03-04T16:17:43.670445+0000","last_peered":"2024-03-04T16:17:43.670445+0000","last_clean":"2024-03-04T16:17:43.670445+0000","last_became_active":"2024-03-04T16:16:53.661564+0000","last_became_peered":"2024-03-04T16:16:53.661564+0000","last_unstale":"2024-03-04T16:17:43.670445+0000","last_undegraded":"2024-03-04T16:17:43.670445+0000","last_fullsized":"2024-03-04T16:17:43.670445+0000","mapping_epoch":296,"log_start":"302'700","ondisk_log_start":"302'700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:53.115414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.327297+0000","last_change":"2024-03-04T16:15:47.547158+0000","last_active":"2024-03-04T16:17:42.327297+0000","last_peered":"2024-03-04T16:17:42.327297+0000","last_clean":"2024-03-04T16:17:42.327297+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:17:42.327297+0000","last_undegraded":"2024-03-04T16:17:42.327297+0000","last_fullsized":"2024-03-04T16:17:42.327297+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:47.547022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:47.547022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":454,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.335795+0000","last_change":"2024-03-04T16:16:07.345839+0000","last_active":"2024-03-04T16:17:42.335795+0000","last_peered":"2024-03-04T16:17:42.335795+0000","last_clean":"2024-03-04T16:17:42.335795+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:17:42.335795+0000","last_undegraded":"2024-03-04T16:17:42.335795+0000","last_fullsized":"2024-03-04T16:17:42.335795+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:16:07.345760+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.345760+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:43.265040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":549,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.347220+0000","last_change":"2024-03-04T16:14:59.774268+0000","last_active":"2024-03-04T16:17:42.347220+0000","last_peered":"2024-03-04T16:17:42.347220+0000","last_clean":"2024-03-04T16:17:42.347220+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:17:42.347220+0000","last_undegraded":"2024-03-04T16:17:42.347220+0000","last_fullsized":"2024-03-04T16:17:42.347220+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.774151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:45.190454+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":589,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-04T16:17:42.326254+0000","last_change":"2024-03-04T16:15:55.587146+0000","last_active":"2024-03-04T16:17:42.326254+0000","last_peered":"2024-03-04T16:17:42.326254+0000","last_clean":"2024-03-04T16:17:42.326254+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:17:42.326254+0000","last_undegraded":"2024-03-04T16:17:42.326254+0000","last_fullsized":"2024-03-04T16:17:42.326254+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:15:55.587052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:55.587052+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4712169472,"num_objects":71902,"num_object_clones":0,"num_object_copies":215706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71902,"num_write_kb":4601728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14116585472,"data_stored":14116585472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65402,"ondisk_log_size":65402,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":892,"num_read_kb":887,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070648,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2293648,"kb_used_data":1698820,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91451504,"statfs":{"total":95995035648,"available":93646340096,"internally_reserved":0,"allocated":1739591680,"data_stored":1739098309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1820520,"kb_used_data":1149344,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91924632,"statfs":{"total":95995035648,"available":94130823168,"internally_reserved":0,"allocated":1176928256,"data_stored":1176412129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3385128,"kb_used_data":2834712,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":90360024,"statfs":{"total":95995035648,"available":92528664576,"internally_reserved":0,"allocated":2902745088,"data_stored":2902238048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411536,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149899,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3107616,"kb_used_data":2286484,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":90637536,"statfs":{"total":95995035648,"available":92812836864,"internally_reserved":0,"allocated":2341359616,"data_stored":2340855817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":840825288},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953670,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4224444,"kb_used_data":3657396,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":89520708,"statfs":{"total":95995035648,"available":91669204992,"internally_reserved":0,"allocated":3745173504,"data_stored":3744694390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":580647365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626626,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2660164,"kb_used_data":2001464,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91084988,"statfs":{"total":95995035648,"available":93271027712,"internally_reserved":0,"allocated":2049499136,"data_stored":2049022529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2955083776,"data_stored":2955083776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1186594816,"data_stored":1186594816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2374828032,"data_stored":2374828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3778740224,"data_stored":3778740224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2067988480,"data_stored":2067988480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1753350144,"data_stored":1753350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:17:51.304 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-04T16:17:51.305 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:17:51.305 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-04T16:17:51.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:17:51.316 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:51.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:51.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:51.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:51.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:51.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:51.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: osdmap e313: 8 total, 8 up, 6 in 2024-03-04T16:17:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: osdmap e313: 8 total, 8 up, 6 in 2024-03-04T16:17:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:51 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:17:51.746 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:51.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: osdmap e313: 8 total, 8 up, 6 in 2024-03-04T16:17:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:51 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:17:51.883 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:51.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:51.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:52.004 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 48 16 84347 84331 109.793 90.6875 0.00644201 0.00910334 2024-03-04T16:17:52.025 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:52.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:52.163 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:52.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:52.300 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:52.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:52.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:52.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:52.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:52.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:17:52.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:17:52.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:17:52.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:17:52.615 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[29779]: pgmap v671: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-04T16:17:52.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:17:52.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:17:52.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:17:52.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:17:52.616 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:52 smithi038 ceph-mon[38113]: pgmap v671: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-04T16:17:52.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:52.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.731 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:52.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:52 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:17:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:52 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:17:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:52 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:17:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:52 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:17:52.861 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:52 smithi176 ceph-mon[33619]: pgmap v671: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.1 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-04T16:17:52.869 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:52.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:52.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:53.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 49 16 85823 85807 109.435 92.25 0.00793353 0.00913382 2024-03-04T16:17:53.008 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:53.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:53.119 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:53.119 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:17:53.141 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:53.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:53.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:53.288 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:53.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:53.420 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:17:53.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:53.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:53.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:17:53.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:17:53.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[29779]: osdmap e314: 8 total, 8 up, 6 in 2024-03-04T16:17:53.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:53.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[38113]: osdmap e314: 8 total, 8 up, 6 in 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:53.531 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:17:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:17:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:17:53.571 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:53.673 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:53.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:53.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:53.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:17:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:17:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-mon[33619]: osdmap e314: 8 total, 8 up, 6 in 2024-03-04T16:17:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:17:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:17:53.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:53.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:53.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:54.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 50 16 87217 87201 108.988 87.125 0.0122987 0.00917085 2024-03-04T16:17:54.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.129 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:54.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:54.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:54.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:54.367 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:54.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[29779]: pgmap v673: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:17:54.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:17:54.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:17:54.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[38113]: pgmap v673: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:17:54.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:54 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:17:54.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:54.549 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:54.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:54.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:54.687 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:54.716 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:17:54.758 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:17:54.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:54.801 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:17:54.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:54.840 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:17:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:54 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:17:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:54 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:17:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:54 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:17:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:54 smithi176 ceph-mon[33619]: pgmap v673: 148 pgs: 148 active+clean; 5.1 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-04T16:17:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:54 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:17:54.880 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:17:54.925 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:17:54.934 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:54.962 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:17:54.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:54.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:55.004 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:17:55.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 51 16 88681 88665 108.645 91.5 0.0123789 0.00919932 2024-03-04T16:17:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:17:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:17:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:17:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:17:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:17:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:17:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:17:55.007 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:17:55.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:55.105 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:55.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:55.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:55.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:55.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:55.390 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:55.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:55.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:55.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:17:55.667 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/340022626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:17:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:17:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:17:55.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/340022626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:55.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:55.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:17:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/340022626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:17:55.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:55.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:55.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:56.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 52 16 90104 90088 108.266 88.9375 0.00850816 0.0092323 2024-03-04T16:17:56.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:56.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:56.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:56.225 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:56.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:56.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:56.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:56.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:56.502 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:56.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:56.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[29779]: pgmap v674: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:17:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:56 smithi038 ceph-mon[38113]: pgmap v674: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-04T16:17:56.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:56.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:56 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:17:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:56 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:17:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:56 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:17:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:56 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:17:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:56 smithi176 ceph-mon[33619]: pgmap v674: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-04T16:17:56.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:56.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:56.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:57.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 53 16 91558 91542 107.938 90.875 0.00876729 0.00926006 2024-03-04T16:17:57.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:17:57.057 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:17:57.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:57.199 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:57.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:17:57.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:57.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:57.539 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:17:57.539 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:17:57.540 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:17:57.540 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:17:57.540 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:17:57.540 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:57 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:17:57.599 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:57.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:57.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:57.766 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:57 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:17:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:57 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:17:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:57 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:17:57.867 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:57.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:57.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:58.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 54 15 92863 92848 107.45 81.625 0.0126102 0.00930215 2024-03-04T16:17:58.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.045 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:58.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:58.192 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:58.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:58.337 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:58.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:17:58.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:17:58.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:58.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:58.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:58.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:17:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:17:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[29779]: pgmap v675: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-04T16:17:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2535844545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[29779]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:17:58.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:17:58.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[38113]: pgmap v675: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-04T16:17:58.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2535844545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:58 smithi038 ceph-mon[38113]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:17:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:58 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:17:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:58 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:17:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:58 smithi176 ceph-mon[33619]: pgmap v675: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 886 op/s 2024-03-04T16:17:58.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2535844545' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:58 smithi176 ceph-mon[33619]: from='client.35663 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:17:58.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:58.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:17:58.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:59.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 55 16 94174 94158 106.985 81.875 0.006376 0.00934215 2024-03-04T16:17:59.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:59.106 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:17:59.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.173 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:59.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:59.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:17:59.314 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:17:59.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:59.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:59.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:17:59.600 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:17:59.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:17:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: osdmap e315: 8 total, 8 up, 6 in 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:17:59.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: osdmap e315: 8 total, 8 up, 6 in 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:17:59.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:17:59 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:17:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.754 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:17:59.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:17:59.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:17:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:17:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:17:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:17:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:17:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: from='client.35663 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: osdmap e315: 8 total, 8 up, 6 in 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:17:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:17:59 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:17:59.904 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:17:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:17:59.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:00.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 56 16 95583 95567 106.647 88.0625 0.0146126 0.00937253 2024-03-04T16:18:00.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:00.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:00.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:18:00.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:00.173 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:00.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.321 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:00.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:00.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.473 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:00.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:00.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:18:00.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:18:00.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:18:00.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:00.607 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:00.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:00.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[29779]: pgmap v677: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[29779]: osdmap e316: 8 total, 8 up, 6 in 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[38113]: pgmap v677: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-04T16:18:00.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:00 smithi038 ceph-mon[38113]: osdmap e316: 8 total, 8 up, 6 in 2024-03-04T16:18:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.753 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:00.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:00.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:00 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:18:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:00 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:18:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:00 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:18:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:00 smithi176 ceph-mon[33619]: pgmap v677: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.77k op/s 2024-03-04T16:18:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:00 smithi176 ceph-mon[33619]: osdmap e316: 8 total, 8 up, 6 in 2024-03-04T16:18:00.893 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:00.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:00.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:00.995 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:01.005 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 57 16 96858 96842 106.174 79.6875 0.00585529 0.0094137 2024-03-04T16:18:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:01.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:01.041 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:18:01.041 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:18:01.041 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:18:01.041 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:18:01.041 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114774 8.6 8:694af726:::benchmark_data_smithi038_107057_object114773:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e193)", 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 452.79730956600002, 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1166456839999999, 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114774 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.042 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229910+0000", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229912+0000", 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2620000000000001e-06 2024-03-04T16:18:01.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229973+0000", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0553000000000001e-05 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.229996+0000", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2646999999999998e-05 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230005+0000", 2024-03-04T16:18:01.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4769999999999992e-06 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230092+0000", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6667000000000002e-05 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230253+0000", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016137700000000001 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 6,7", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.230416+0000", 2024-03-04T16:18:01.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016295200000000001 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243867+0000", 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013450632000000001 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:28.243882+0000", 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5690999999999999e-05 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232356+0000", 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98847337300000004 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.232375+0000", 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9768000000000001e-05 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.330696+0000", 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.098320972 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for degraded object", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.331886+0000", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011898729999999999 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346472+0000", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014585653000000001 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346556+0000", 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3787000000000003e-05 2024-03-04T16:18:01.048 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114798 8.6 8:6e193aa2:::benchmark_data_smithi038_107057_object114797:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 451.575101781, 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.89634460400000004, 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114798 2024-03-04T16:18:01.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452118+0000", 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.050 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452121+0000", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.022e-06 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452154+0000", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2851000000000001e-05 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452177+0000", 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2880000000000001e-05 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452185+0000", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8149999999999995e-06 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452277+0000", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1267999999999994e-05 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.452292+0000", 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5811000000000002e-05 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332203+0000", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87991063700000005 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332343+0000", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013942199999999999 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.332826+0000", 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048315199999999998 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346595+0000", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013768927 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346632+0000", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7630999999999999e-05 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346680+0000", 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7589999999999997e-05 2024-03-04T16:18:01.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348078+0000", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013984640000000001 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348149+0000", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0673000000000006e-05 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348462+0000", 2024-03-04T16:18:01.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031346100000000001 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:114804 8.6 8:642351e4:::benchmark_data_smithi038_107057_object114803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 451.55347679599998, 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87523897799999995, 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:18:01.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 114804 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473743+0000", 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473746+0000", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.8600000000000001e-06 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473780+0000", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4461000000000001e-05 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473798+0000", 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8297000000000001e-05 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473809+0000", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883999999999999e-05 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473890+0000", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0513999999999995e-05 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:29.473913+0000", 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3564e-05 2024-03-04T16:18:01.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343022+0000", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.86910901200000001 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343537+0000", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00051479200000000003 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6,7", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.343752+0000", 2024-03-04T16:18:01.060 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000215195 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.346685+0000", 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.002932933 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.347988+0000", 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0013024550000000001 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.061 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348035+0000", 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6859e-05 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348780+0000", 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00074556000000000004 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348845+0000", 2024-03-04T16:18:01.062 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4453000000000001e-05 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:10:30.348982+0000", 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013713899999999999 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.15171.0:207574 8.6 8:63054e5a:::benchmark_data_smithi038_107057_object85939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e227)", 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 354.52533392200002, 2024-03-04T16:18:01.063 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.183928063, 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.15171", 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3341984358", 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 207574 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501886+0000", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.501887+0000", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.246e-06 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503000+0000", 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001113459 2024-03-04T16:18:01.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503016+0000", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5690000000000001e-05 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.503024+0000", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3299999999999999e-06 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507462+0000", 2024-03-04T16:18:01.066 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044379500000000004 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507549+0000", 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.6785000000000003e-05 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.507678+0000", 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012845599999999999 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.067 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.509750+0000", 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020719219999999999 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.510463+0000", 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000713779 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685583+0000", 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.17511929700000001 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.068 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685760+0000", 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177682 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:12:06.685814+0000", 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3467000000000002e-05 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.069 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:73133 10.4 10:2a9f31f9:::benchmark_data_smithi038_155391_object73132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e312)", 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:44.096104+0000", 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 16.931115260999999, 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038758894000000002, 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 73133 2024-03-04T16:18:01.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096104+0000", 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096104+0000", 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.071 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096106+0000", 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2759999999999999e-06 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096156+0000", 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0278999999999998e-05 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096164+0000", 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3489999999999992e-06 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096171+0000", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5749999999999997e-06 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.096995+0000", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000823706 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.097123+0000", 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012835200000000001 2024-03-04T16:18:01.073 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4", 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.097256+0000", 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013355999999999999 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.102560+0000", 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0053038019999999998 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.074 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.103248+0000", 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000687344 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.134762+0000", 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.031514721000000002 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.134820+0000", 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7951000000000001e-05 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.075 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:44.134863+0000", 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2979000000000003e-05 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:89058 10.4 10:2154c4a0:::benchmark_data_smithi038_155391_object89057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.258643+0000", 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7685761690000001, 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.072513261999999995, 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 89058 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258643+0000", 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.077 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258643+0000", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258645+0000", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0809999999999999e-06 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258676+0000", 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1426e-05 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.078 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258685+0000", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4159999999999993e-06 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258692+0000", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.126e-06 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.258771+0000", 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8892000000000003e-05 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.079 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259190+0000", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00041823799999999998 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259688+0000", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00049800500000000004 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330728+0000", 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.071040395000000006 2024-03-04T16:18:01.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330774+0000", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5701000000000002e-05 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330812+0000", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8257999999999999e-05 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331026+0000", 2024-03-04T16:18:01.081 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000213606 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331157+0000", 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131118 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.082 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:89062 10.4 10:28cbc18b:::benchmark_data_smithi038_155391_object89061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.259587+0000", 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7676328730000002, 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.071931765999999994, 2024-03-04T16:18:01.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 89062 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259587+0000", 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.084 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259587+0000", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259588+0000", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9199999999999999e-07 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259609+0000", 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1313000000000001e-05 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.085 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259620+0000", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0815999999999999e-05 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.259627+0000", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.729e-06 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263161+0000", 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035337379999999998 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.086 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263289+0000", 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000127377 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4", 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263449+0000", 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016066600000000001 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.087 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330734+0000", 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067284503999999995 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331220+0000", 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000486169 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.088 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331265+0000", 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5491999999999997e-05 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331428+0000", 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016236000000000001 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331519+0000", 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0710000000000004e-05 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.089 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:89056 10.b e314/296 10:d7be6672:::benchmark_data_smithi038_155391_object89055:head v 314'5568, mlcod=314'5564)", 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.261516+0000", 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7657032470000003, 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.069659566000000006, 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.090 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261516+0000", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261516+0000", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261518+0000", 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4959999999999999e-06 2024-03-04T16:18:01.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261541+0000", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2830999999999999e-05 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261550+0000", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6520000000000001e-06 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261557+0000", 2024-03-04T16:18:01.092 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.489e-06 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261652+0000", 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5626999999999998e-05 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.261671+0000", 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8244e-05 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.093 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331134+0000", 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.069463548 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331176+0000", 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1678999999999999e-05 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:89067 10.4 10:2805172e:::benchmark_data_smithi038_155391_object89066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.262955+0000", 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7642646190000004, 2024-03-04T16:18:01.094 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.077356624999999998, 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 89067 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.262955+0000", 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.095 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.262955+0000", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.262957+0000", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0669999999999999e-06 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.262981+0000", 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.4372999999999999e-05 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.262992+0000", 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.095e-05 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.097 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263042+0000", 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9196999999999997e-05 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.098 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328128+0000", 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.065086884999999997 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328279+0000", 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015094500000000001 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4", 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328458+0000", 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017865200000000001 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.099 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331568+0000", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.003109494 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339572+0000", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0080046760000000005 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340225+0000", 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00065246399999999995 2024-03-04T16:18:01.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340272+0000", 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7021000000000003e-05 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340312+0000", 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.9901000000000003e-05 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.101 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:89063 10.6 e314/296 10:6cf36c38:::benchmark_data_smithi038_155391_object89062:head v 314'5634, mlcod=314'5630)", 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.263342+0000", 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7638780450000002, 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.065527245999999997, 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263342+0000", 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.102 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263342+0000", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263343+0000", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.327e-06 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263372+0000", 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9561e-05 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.103 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263381+0000", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9500000000000007e-06 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263389+0000", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6569999999999997e-06 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263469+0000", 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9751000000000003e-05 2024-03-04T16:18:01.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263489+0000", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9734000000000001e-05 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328763+0000", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.065274483999999994 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328869+0000", 2024-03-04T16:18:01.105 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000105782 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.35585.0:89058 10.4 e314/296 ondisk, result = 0)", 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.263725+0000", 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7634948240000003, 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067056128000000007, 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.106 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263725+0000", 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263725+0000", 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263726+0000", 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1170000000000001e-06 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.107 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263730+0000", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5449999999999997e-06 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263739+0000", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0240000000000003e-06 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.263746+0000", 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3620000000000002e-06 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.108 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330761+0000", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.067014780999999995 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330768+0000", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1749999999999999e-06 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.330781+0000", 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3124e-05 2024-03-04T16:18:01.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:89066 10.b e314/296 10:d0dc88ff:::benchmark_data_smithi038_155391_object89065:head v 314'5569, mlcod=314'5565)", 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.264031+0000", 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.76318865, 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.075873854000000004, 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264031+0000", 2024-03-04T16:18:01.110 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264031+0000", 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264032+0000", 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0470000000000001e-06 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.111 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264055+0000", 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3405999999999999e-05 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264063+0000", 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.943e-06 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264069+0000", 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0499999999999997e-06 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.112 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327786+0000", 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.063716430000000004 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327817+0000", 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.0954000000000002e-05 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339666+0000", 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011849505 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339905+0000", 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000238519 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.114 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:89068 10.a 10:50f50e75:::benchmark_data_smithi038_155391_object89067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.264277+0000", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7629426859999997, 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.076219007000000005, 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 89068 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.115 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264277+0000", 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264277+0000", 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264278+0000", 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3510000000000001e-06 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.116 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.264300+0000", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2203000000000001e-05 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327877+0000", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.063576143000000002 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327892+0000", 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5735000000000001e-05 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.117 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328110+0000", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00021754599999999999 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328280+0000", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016997 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328458+0000", 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017809799999999999 2024-03-04T16:18:01.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.337778+0000", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0093200709999999992 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339587+0000", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001809134 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340198+0000", 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00061053400000000003 2024-03-04T16:18:01.119 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340249+0000", 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0847999999999998e-05 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.340496+0000", 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024737400000000002 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.120 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop_reply(client.35585.0:89062 10.4 e314/296 ondisk, result = 0)", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.266121+0000", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7610981160000003, 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.065109073000000003, 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "started", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266121+0000", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.121 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266121+0000", 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266122+0000", 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8700000000000004e-07 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266127+0000", 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4139999999999996e-06 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.122 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328567+0000", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.062439999000000003 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328578+0000", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0868e-05 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331193+0000", 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026155420000000002 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.123 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331214+0000", 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1160000000000001e-05 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.331231+0000", 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6103000000000002e-05 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.124 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:89072 10.2 e314/296 10:4b8fcf9d:::benchmark_data_smithi038_155391_object89071:head v 314'5609, mlcod=314'5607)", 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:55.266264+0000", 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 5.7609555859999997, 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.073052371000000005, 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.125 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266264+0000", 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266264+0000", 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.126 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266265+0000", 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3480000000000001e-06 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.266291+0000", 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5194999999999998e-05 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.127 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327861+0000", 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.061570179000000003 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.327875+0000", 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4215e-05 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.128 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328236+0000", 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00036117300000000001 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.328257+0000", 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1276000000000001e-05 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339254+0000", 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010996789 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.129 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:55.339316+0000", 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2195999999999999e-05 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_repop(client.35585.0:90692 10.6 e314/296 10:630572e3:::benchmark_data_smithi038_155391_object90691:head v 314'5728, mlcod=314'5725)", 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:56.390420+0000", 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.6367992569999998, 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.039072469999999998, 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.130 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390420+0000", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390420+0000", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390421+0000", 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.052e-06 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.131 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390476+0000", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4873000000000001e-05 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390484+0000", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0229999999999992e-06 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390490+0000", 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2599999999999996e-06 2024-03-04T16:18:01.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390536+0000", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6578e-05 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.390554+0000", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8224999999999999e-05 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.429440+0000", 2024-03-04T16:18:01.133 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.038885231999999999 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:56.429493+0000", 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3226999999999997e-05 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:93444 10.a 10:578410b1:::benchmark_data_smithi038_155391_object93443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:58.438277+0000", 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.588942657, 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.046634373999999999, 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 93444 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.135 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438277+0000", 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438277+0000", 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.136 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438279+0000", 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.573e-06 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438301+0000", 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.264e-05 2024-03-04T16:18:01.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438311+0000", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7180000000000008e-06 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438318+0000", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4760000000000001e-06 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438359+0000", 2024-03-04T16:18:01.138 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0839e-05 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438493+0000", 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013360100000000001 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438658+0000", 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000164795 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.139 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.442077+0000", 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0034194350000000002 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.443086+0000", 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010092910000000001 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.484774+0000", 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.041687308999999999 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.140 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.484843+0000", 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8998999999999994e-05 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.484911+0000", 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8697999999999999e-05 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:93445 10.4 10:2528ff41:::benchmark_data_smithi038_155391_object93444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.141 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:58.438446+0000", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.588773239, 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.052856067, 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 93445 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438446+0000", 2024-03-04T16:18:01.142 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438446+0000", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438447+0000", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8700000000000004e-07 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438488+0000", 2024-03-04T16:18:01.143 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0373000000000001e-05 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438496+0000", 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7690000000000002e-06 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.144 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438503+0000", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.8789999999999997e-06 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438550+0000", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6437999999999998e-05 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438667+0000", 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116888 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.145 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.438804+0000", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013757900000000001 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.442777+0000", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0039727709999999999 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444015+0000", 2024-03-04T16:18:01.146 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012381619999999999 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491199+0000", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047183999999999997 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491258+0000", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9032e-05 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491302+0000", 2024-03-04T16:18:01.147 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4189e-05 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:93454 10.5 10:ad1f1c42:::benchmark_data_smithi038_155391_object93453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:58.444263+0000", 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5829566809999998, 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.047250214999999998, 2024-03-04T16:18:01.148 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 93454 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444263+0000", 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.149 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444263+0000", 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.150 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444265+0000", 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6309999999999999e-06 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444283+0000", 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7978999999999998e-05 2024-03-04T16:18:01.151 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444291+0000", 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8419999999999994e-06 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.152 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.153 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.153 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444297+0000", 2024-03-04T16:18:01.153 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3539999999999999e-06 2024-03-04T16:18:01.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.153 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.154 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.154 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444345+0000", 2024-03-04T16:18:01.154 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7855999999999997e-05 2024-03-04T16:18:01.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.154 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.155 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.155 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444439+0000", 2024-03-04T16:18:01.155 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4025999999999997e-05 2024-03-04T16:18:01.155 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.444563+0000", 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012397800000000001 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.156 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.485293+0000", 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.040730205999999998 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.157 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.485459+0000", 2024-03-04T16:18:01.158 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00016610400000000001 2024-03-04T16:18:01.158 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.158 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.158 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.158 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491430+0000", 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0059709480000000002 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491477+0000", 2024-03-04T16:18:01.159 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6783000000000003e-05 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491513+0000", 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6507999999999997e-05 2024-03-04T16:18:01.160 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:93457 10.5 10:a3573f86:::benchmark_data_smithi038_155391_object93456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-03-04T16:18:01.161 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:17:58.445683+0000", 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 2.5815361819999998, 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.053171568000000002, 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:18:01.162 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 93457 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.163 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445683+0000", 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:18:01.164 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445683+0000", 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445685+0000", 2024-03-04T16:18:01.165 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.522e-06 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445705+0000", 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0254999999999998e-05 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.166 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445713+0000", 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2390000000000002e-06 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.167 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.445720+0000", 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4300000000000003e-06 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:18:01.168 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.446445+0000", 2024-03-04T16:18:01.169 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00072555000000000004 2024-03-04T16:18:01.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.169 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.169 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:18:01.169 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.446529+0000", 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3132000000000003e-05 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.481313+0000", 2024-03-04T16:18:01.170 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.034784164999999999 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.488521+0000", 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0072084159999999996 2024-03-04T16:18:01.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.491390+0000", 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0028687389999999999 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.172 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.498763+0000", 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.007373597 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.173 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.498816+0000", 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2924999999999999e-05 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:18:01.174 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:17:58.498855+0000", 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8597999999999999e-05 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:18:01.175 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:18:01.176 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:18:01.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:01.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:01.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:01.362 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:01.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:01.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:01.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:01.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:01.630 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:18:01.635 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:18:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.649 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:01.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:01.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:01 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:18:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:01 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:18:01.743 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:01 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:18:01.750 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:01.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:01.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:18:01.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:01.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:01.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:01.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:02.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 58 16 98154 98138 105.74 81 0.0159278 0.00945268 2024-03-04T16:18:02.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.055 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:02.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:02.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:02.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.468 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:02.569 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:02.606 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:02.684 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":678,"stamp":"2024-03-04T16:18:00.323478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6107699550,"num_objects":93385,"num_object_clones":0,"num_object_copies":280155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93385,"num_whiteouts":0,"num_read":2255,"num_read_kb":2101,"num_write":94088,"num_write_kb":5966429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":265,"num_bytes_recovered":3550482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66654,"ondisk_log_size":66654,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":21735700,"kb_used_data":17753060,"kb_used_omap":9,"kb_used_meta":3982582,"kb_avail":540735212,"statfs":{"total":575970213888,"available":553712857088,"internally_reserved":0,"allocated":18179133440,"data_stored":18176081528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":4078164647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,3,6,3,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":926875648,"num_objects":14143,"num_object_clones":0,"num_object_copies":42429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14143,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":14143,"num_write_kb":905152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.905941"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":545,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.425936+0000","last_change":"2024-03-04T16:16:48.504644+0000","last_active":"2024-03-04T16:17:52.425936+0000","last_peered":"2024-03-04T16:17:52.425936+0000","last_clean":"2024-03-04T16:17:52.425936+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:17:52.425936+0000","last_undegraded":"2024-03-04T16:17:52.425936+0000","last_fullsized":"2024-03-04T16:17:52.425936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:48.504543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:16:48.504543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.379284+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":551,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489262+0000","last_change":"2024-03-04T16:17:48.323042+0000","last_active":"2024-03-04T16:17:58.489262+0000","last_peered":"2024-03-04T16:17:58.489262+0000","last_clean":"2024-03-04T16:17:58.489262+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:17:58.489262+0000","last_undegraded":"2024-03-04T16:17:58.489262+0000","last_fullsized":"2024-03-04T16:17:58.489262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.322908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.322908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:04.615370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":415,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.496731+0000","last_change":"2024-03-04T16:17:47.623707+0000","last_active":"2024-03-04T16:17:58.496731+0000","last_peered":"2024-03-04T16:17:58.496731+0000","last_clean":"2024-03-04T16:17:58.496731+0000","last_became_active":"2024-03-04T16:10:29.339335+0000","last_became_peered":"2024-03-04T16:10:29.339335+0000","last_unstale":"2024-03-04T16:17:58.496731+0000","last_undegraded":"2024-03-04T16:17:58.496731+0000","last_fullsized":"2024-03-04T16:17:58.496731+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.623578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.623578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:10.327564+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":594,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.425527+0000","last_change":"2024-03-04T16:17:07.218989+0000","last_active":"2024-03-04T16:17:52.425527+0000","last_peered":"2024-03-04T16:17:52.425527+0000","last_clean":"2024-03-04T16:17:52.425527+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:17:52.425527+0000","last_undegraded":"2024-03-04T16:17:52.425527+0000","last_fullsized":"2024-03-04T16:17:52.425527+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:17:07.218848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.218848+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.217883+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":545,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.492298+0000","last_change":"2024-03-04T16:16:55.868681+0000","last_active":"2024-03-04T16:17:58.492298+0000","last_peered":"2024-03-04T16:17:58.492298+0000","last_clean":"2024-03-04T16:17:58.492298+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:17:58.492298+0000","last_undegraded":"2024-03-04T16:17:58.492298+0000","last_fullsized":"2024-03-04T16:17:58.492298+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.868581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.868581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:24.049066+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":547,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.425639+0000","last_change":"2024-03-04T16:15:44.517042+0000","last_active":"2024-03-04T16:17:52.425639+0000","last_peered":"2024-03-04T16:17:52.425639+0000","last_clean":"2024-03-04T16:17:52.425639+0000","last_became_active":"2024-03-04T16:10:29.322303+0000","last_became_peered":"2024-03-04T16:10:29.322303+0000","last_unstale":"2024-03-04T16:17:52.425639+0000","last_undegraded":"2024-03-04T16:17:52.425639+0000","last_fullsized":"2024-03-04T16:17:52.425639+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:44.516926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:44.516926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":556,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.488811+0000","last_change":"2024-03-04T16:16:07.030698+0000","last_active":"2024-03-04T16:17:58.488811+0000","last_peered":"2024-03-04T16:17:58.488811+0000","last_clean":"2024-03-04T16:17:58.488811+0000","last_became_active":"2024-03-04T16:10:29.251400+0000","last_became_peered":"2024-03-04T16:10:29.251400+0000","last_unstale":"2024-03-04T16:17:58.488811+0000","last_undegraded":"2024-03-04T16:17:58.488811+0000","last_fullsized":"2024-03-04T16:17:58.488811+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.030580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.030580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":656,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.488821+0000","last_change":"2024-03-04T16:15:51.042206+0000","last_active":"2024-03-04T16:17:58.488821+0000","last_peered":"2024-03-04T16:17:58.488821+0000","last_clean":"2024-03-04T16:17:58.488821+0000","last_became_active":"2024-03-04T16:10:17.470722+0000","last_became_peered":"2024-03-04T16:10:17.470722+0000","last_unstale":"2024-03-04T16:17:58.488821+0000","last_undegraded":"2024-03-04T16:17:58.488821+0000","last_fullsized":"2024-03-04T16:17:58.488821+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:15:51.042010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.042010+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":403,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.497627+0000","last_change":"2024-03-04T16:16:00.432293+0000","last_active":"2024-03-04T16:17:58.497627+0000","last_peered":"2024-03-04T16:17:58.497627+0000","last_clean":"2024-03-04T16:17:58.497627+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:17:58.497627+0000","last_undegraded":"2024-03-04T16:17:58.497627+0000","last_fullsized":"2024-03-04T16:17:58.497627+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.432170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:07:36.929522+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.432170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":548,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430372+0000","last_change":"2024-03-04T16:17:03.584844+0000","last_active":"2024-03-04T16:17:52.430372+0000","last_peered":"2024-03-04T16:17:52.430372+0000","last_clean":"2024-03-04T16:17:52.430372+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:17:52.430372+0000","last_undegraded":"2024-03-04T16:17:52.430372+0000","last_fullsized":"2024-03-04T16:17:52.430372+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.584766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.584766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:14.913502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":570,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430653+0000","last_change":"2024-03-04T16:16:18.732834+0000","last_active":"2024-03-04T16:17:52.430653+0000","last_peered":"2024-03-04T16:17:52.430653+0000","last_clean":"2024-03-04T16:17:52.430653+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:17:52.430653+0000","last_undegraded":"2024-03-04T16:17:52.430653+0000","last_fullsized":"2024-03-04T16:17:52.430653+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.732730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.732730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":614,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426166+0000","last_change":"2024-03-04T16:17:49.277087+0000","last_active":"2024-03-04T16:17:52.426166+0000","last_peered":"2024-03-04T16:17:52.426166+0000","last_clean":"2024-03-04T16:17:52.426166+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:17:52.426166+0000","last_undegraded":"2024-03-04T16:17:52.426166+0000","last_fullsized":"2024-03-04T16:17:52.426166+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:49.276987+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.276987+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:19.171525+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430020+0000","last_change":"2024-03-04T16:16:09.872300+0000","last_active":"2024-03-04T16:17:52.430020+0000","last_peered":"2024-03-04T16:17:52.430020+0000","last_clean":"2024-03-04T16:17:52.430020+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:17:52.430020+0000","last_undegraded":"2024-03-04T16:17:52.430020+0000","last_fullsized":"2024-03-04T16:17:52.430020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:16:09.872152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424006+0000","last_change":"2024-03-04T16:16:49.431749+0000","last_active":"2024-03-04T16:17:52.424006+0000","last_peered":"2024-03-04T16:17:52.424006+0000","last_clean":"2024-03-04T16:17:52.424006+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:17:52.424006+0000","last_undegraded":"2024-03-04T16:17:52.424006+0000","last_fullsized":"2024-03-04T16:17:52.424006+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:49.431643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.431643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:54.959231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430243+0000","last_change":"2024-03-04T16:16:06.843357+0000","last_active":"2024-03-04T16:17:52.430243+0000","last_peered":"2024-03-04T16:17:52.430243+0000","last_clean":"2024-03-04T16:17:52.430243+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:17:52.430243+0000","last_undegraded":"2024-03-04T16:17:52.430243+0000","last_fullsized":"2024-03-04T16:17:52.430243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:06.843231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:06.843231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":592,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:54.411883+0000","last_change":"2024-03-04T16:17:54.411929+0000","last_active":"2024-03-04T16:17:54.411883+0000","last_peered":"2024-03-04T16:17:54.411883+0000","last_clean":"2024-03-04T16:17:54.411883+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:17:54.411883+0000","last_undegraded":"2024-03-04T16:17:54.411883+0000","last_fullsized":"2024-03-04T16:17:54.411883+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-03-04T16:17:54.411724+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.411724+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:13.954367+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":544,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430960+0000","last_change":"2024-03-04T16:15:43.560939+0000","last_active":"2024-03-04T16:17:52.430960+0000","last_peered":"2024-03-04T16:17:52.430960+0000","last_clean":"2024-03-04T16:17:52.430960+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:17:52.430960+0000","last_undegraded":"2024-03-04T16:17:52.430960+0000","last_fullsized":"2024-03-04T16:17:52.430960+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:43.560826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:43.560826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490292+0000","last_change":"2024-03-04T16:17:49.317278+0000","last_active":"2024-03-04T16:17:58.490292+0000","last_peered":"2024-03-04T16:17:58.490292+0000","last_clean":"2024-03-04T16:17:58.490292+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:17:58.490292+0000","last_undegraded":"2024-03-04T16:17:58.490292+0000","last_fullsized":"2024-03-04T16:17:58.490292+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:49.317160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.317160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:04.256840+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":395,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495603+0000","last_change":"2024-03-04T16:16:03.471410+0000","last_active":"2024-03-04T16:17:58.495603+0000","last_peered":"2024-03-04T16:17:58.495603+0000","last_clean":"2024-03-04T16:17:58.495603+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:17:58.495603+0000","last_undegraded":"2024-03-04T16:17:58.495603+0000","last_fullsized":"2024-03-04T16:17:58.495603+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.471180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.471180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"286'34","reported_seq":372,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489911+0000","last_change":"2024-03-04T16:16:16.420065+0000","last_active":"2024-03-04T16:17:58.489911+0000","last_peered":"2024-03-04T16:17:58.489911+0000","last_clean":"2024-03-04T16:17:58.489911+0000","last_became_active":"2024-03-04T16:10:17.493356+0000","last_became_peered":"2024-03-04T16:10:17.493356+0000","last_unstale":"2024-03-04T16:17:58.489911+0000","last_undegraded":"2024-03-04T16:17:58.489911+0000","last_fullsized":"2024-03-04T16:17:58.489911+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.419895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":546,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491871+0000","last_change":"2024-03-04T16:17:48.459727+0000","last_active":"2024-03-04T16:17:58.491871+0000","last_peered":"2024-03-04T16:17:58.491871+0000","last_clean":"2024-03-04T16:17:58.491871+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:17:58.491871+0000","last_undegraded":"2024-03-04T16:17:58.491871+0000","last_fullsized":"2024-03-04T16:17:58.491871+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.459594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.459594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:09.201445+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":555,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426912+0000","last_change":"2024-03-04T16:16:52.414765+0000","last_active":"2024-03-04T16:17:52.426912+0000","last_peered":"2024-03-04T16:17:52.426912+0000","last_clean":"2024-03-04T16:17:52.426912+0000","last_became_active":"2024-03-04T16:10:17.458973+0000","last_became_peered":"2024-03-04T16:10:17.458973+0000","last_unstale":"2024-03-04T16:17:52.426912+0000","last_undegraded":"2024-03-04T16:17:52.426912+0000","last_fullsized":"2024-03-04T16:17:52.426912+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.414689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.414689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:53.722924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":563,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426352+0000","last_change":"2024-03-04T16:16:17.538604+0000","last_active":"2024-03-04T16:17:52.426352+0000","last_peered":"2024-03-04T16:17:52.426352+0000","last_clean":"2024-03-04T16:17:52.426352+0000","last_became_active":"2024-03-04T16:10:17.480514+0000","last_became_peered":"2024-03-04T16:10:17.480514+0000","last_unstale":"2024-03-04T16:17:52.426352+0000","last_undegraded":"2024-03-04T16:17:52.426352+0000","last_fullsized":"2024-03-04T16:17:52.426352+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:17.538492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.538492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":587,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490733+0000","last_change":"2024-03-04T16:17:03.702758+0000","last_active":"2024-03-04T16:17:58.490733+0000","last_peered":"2024-03-04T16:17:58.490733+0000","last_clean":"2024-03-04T16:17:58.490733+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:17:58.490733+0000","last_undegraded":"2024-03-04T16:17:58.490733+0000","last_fullsized":"2024-03-04T16:17:58.490733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:17:03.702661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.702661+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.888478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428925+0000","last_change":"2024-03-04T16:16:08.368380+0000","last_active":"2024-03-04T16:17:52.428925+0000","last_peered":"2024-03-04T16:17:52.428925+0000","last_clean":"2024-03-04T16:17:52.428925+0000","last_became_active":"2024-03-04T16:10:17.464727+0000","last_became_peered":"2024-03-04T16:10:17.464727+0000","last_unstale":"2024-03-04T16:17:52.428925+0000","last_undegraded":"2024-03-04T16:17:52.428925+0000","last_fullsized":"2024-03-04T16:17:52.428925+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:16:08.368298+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.368298+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"195'10","reported_seq":557,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429761+0000","last_change":"2024-03-04T16:17:08.205355+0000","last_active":"2024-03-04T16:17:52.429761+0000","last_peered":"2024-03-04T16:17:52.429761+0000","last_clean":"2024-03-04T16:17:52.429761+0000","last_became_active":"2024-03-04T16:10:29.321703+0000","last_became_peered":"2024-03-04T16:10:29.321703+0000","last_unstale":"2024-03-04T16:17:52.429761+0000","last_undegraded":"2024-03-04T16:17:52.429761+0000","last_fullsized":"2024-03-04T16:17:52.429761+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'10","last_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:17:08.205209+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.803342+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489508+0000","last_change":"2024-03-04T16:16:15.395144+0000","last_active":"2024-03-04T16:17:58.489508+0000","last_peered":"2024-03-04T16:17:58.489508+0000","last_clean":"2024-03-04T16:17:58.489508+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:17:58.489508+0000","last_undegraded":"2024-03-04T16:17:58.489508+0000","last_fullsized":"2024-03-04T16:17:58.489508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:15.395015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.395015+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":431,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.496024+0000","last_change":"2024-03-04T16:16:08.556672+0000","last_active":"2024-03-04T16:17:58.496024+0000","last_peered":"2024-03-04T16:17:58.496024+0000","last_clean":"2024-03-04T16:17:58.496024+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:17:58.496024+0000","last_undegraded":"2024-03-04T16:17:58.496024+0000","last_fullsized":"2024-03-04T16:17:58.496024+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:08.553910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.553910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":547,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490436+0000","last_change":"2024-03-04T16:17:03.492110+0000","last_active":"2024-03-04T16:17:58.490436+0000","last_peered":"2024-03-04T16:17:58.490436+0000","last_clean":"2024-03-04T16:17:58.490436+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:17:58.490436+0000","last_undegraded":"2024-03-04T16:17:58.490436+0000","last_fullsized":"2024-03-04T16:17:58.490436+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:28.381622+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":73,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495662+0000","last_change":"2024-03-04T16:17:00.852789+0000","last_active":"2024-03-04T16:17:58.495662+0000","last_peered":"2024-03-04T16:17:58.495662+0000","last_clean":"2024-03-04T16:17:58.495662+0000","last_became_active":"2024-03-04T16:16:17.961588+0000","last_became_peered":"2024-03-04T16:16:17.961588+0000","last_unstale":"2024-03-04T16:17:58.495662+0000","last_undegraded":"2024-03-04T16:17:58.495662+0000","last_fullsized":"2024-03-04T16:17:58.495662+0000","mapping_epoch":287,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:00.852665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.852665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:11.304309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":563,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.483938+0000","last_change":"2024-03-04T16:16:08.076532+0000","last_active":"2024-03-04T16:17:58.483938+0000","last_peered":"2024-03-04T16:17:58.483938+0000","last_clean":"2024-03-04T16:17:58.483938+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:17:58.483938+0000","last_undegraded":"2024-03-04T16:17:58.483938+0000","last_fullsized":"2024-03-04T16:17:58.483938+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.076402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":339,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430864+0000","last_change":"2024-03-04T16:17:06.624302+0000","last_active":"2024-03-04T16:17:52.430864+0000","last_peered":"2024-03-04T16:17:52.430864+0000","last_clean":"2024-03-04T16:17:52.430864+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:17:52.430864+0000","last_undegraded":"2024-03-04T16:17:52.430864+0000","last_fullsized":"2024-03-04T16:17:52.430864+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:06.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.624154+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:34.959596+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":559,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424148+0000","last_change":"2024-03-04T16:16:19.400006+0000","last_active":"2024-03-04T16:17:52.424148+0000","last_peered":"2024-03-04T16:17:52.424148+0000","last_clean":"2024-03-04T16:17:52.424148+0000","last_became_active":"2024-03-04T16:10:17.457544+0000","last_became_peered":"2024-03-04T16:10:17.457544+0000","last_unstale":"2024-03-04T16:17:52.424148+0000","last_undegraded":"2024-03-04T16:17:52.424148+0000","last_fullsized":"2024-03-04T16:17:52.424148+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.399876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.399876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":269,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428161+0000","last_change":"2024-03-04T16:15:58.573709+0000","last_active":"2024-03-04T16:17:52.428161+0000","last_peered":"2024-03-04T16:17:52.428161+0000","last_clean":"2024-03-04T16:17:52.428161+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:17:52.428161+0000","last_undegraded":"2024-03-04T16:17:52.428161+0000","last_fullsized":"2024-03-04T16:17:52.428161+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.573599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.573599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":395,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.496669+0000","last_change":"2024-03-04T16:15:58.418770+0000","last_active":"2024-03-04T16:17:58.496669+0000","last_peered":"2024-03-04T16:17:58.496669+0000","last_clean":"2024-03-04T16:17:58.496669+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:17:58.496669+0000","last_undegraded":"2024-03-04T16:17:58.496669+0000","last_fullsized":"2024-03-04T16:17:58.496669+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.418646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.418646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":571,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.484119+0000","last_change":"2024-03-04T16:16:59.675333+0000","last_active":"2024-03-04T16:17:58.484119+0000","last_peered":"2024-03-04T16:17:58.484119+0000","last_clean":"2024-03-04T16:17:58.484119+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:17:58.484119+0000","last_undegraded":"2024-03-04T16:17:58.484119+0000","last_fullsized":"2024-03-04T16:17:58.484119+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:16:59.675176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.675176+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:25.451412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":549,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491919+0000","last_change":"2024-03-04T16:15:09.840395+0000","last_active":"2024-03-04T16:17:58.491919+0000","last_peered":"2024-03-04T16:17:58.491919+0000","last_clean":"2024-03-04T16:17:58.491919+0000","last_became_active":"2024-03-04T16:10:29.250647+0000","last_became_peered":"2024-03-04T16:10:29.250647+0000","last_unstale":"2024-03-04T16:17:58.491919+0000","last_undegraded":"2024-03-04T16:17:58.491919+0000","last_fullsized":"2024-03-04T16:17:58.491919+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:09.840155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:15:09.840155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":555,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429164+0000","last_change":"2024-03-04T16:16:03.398326+0000","last_active":"2024-03-04T16:17:52.429164+0000","last_peered":"2024-03-04T16:17:52.429164+0000","last_clean":"2024-03-04T16:17:52.429164+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:17:52.429164+0000","last_undegraded":"2024-03-04T16:17:52.429164+0000","last_fullsized":"2024-03-04T16:17:52.429164+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:03.398160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:03.398160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":564,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431018+0000","last_change":"2024-03-04T16:17:01.618230+0000","last_active":"2024-03-04T16:17:52.431018+0000","last_peered":"2024-03-04T16:17:52.431018+0000","last_clean":"2024-03-04T16:17:52.431018+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:17:52.431018+0000","last_undegraded":"2024-03-04T16:17:52.431018+0000","last_fullsized":"2024-03-04T16:17:52.431018+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.618146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.618146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.383599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":588,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.488712+0000","last_change":"2024-03-04T16:15:01.756693+0000","last_active":"2024-03-04T16:17:58.488712+0000","last_peered":"2024-03-04T16:17:58.488712+0000","last_clean":"2024-03-04T16:17:58.488712+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:17:58.488712+0000","last_undegraded":"2024-03-04T16:17:58.488712+0000","last_fullsized":"2024-03-04T16:17:58.488712+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:01.756601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:01.756601+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":395,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.497727+0000","last_change":"2024-03-04T16:16:16.701023+0000","last_active":"2024-03-04T16:17:58.497727+0000","last_peered":"2024-03-04T16:17:58.497727+0000","last_clean":"2024-03-04T16:17:58.497727+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:17:58.497727+0000","last_undegraded":"2024-03-04T16:17:58.497727+0000","last_fullsized":"2024-03-04T16:17:58.497727+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.700854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.700854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":553,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490745+0000","last_change":"2024-03-04T16:16:05.072152+0000","last_active":"2024-03-04T16:17:58.490745+0000","last_peered":"2024-03-04T16:17:58.490745+0000","last_clean":"2024-03-04T16:17:58.490745+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:17:58.490745+0000","last_undegraded":"2024-03-04T16:17:58.490745+0000","last_fullsized":"2024-03-04T16:17:58.490745+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:05.071956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:05.071956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":565,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:56.238538+0000","last_change":"2024-03-04T16:17:56.238575+0000","last_active":"2024-03-04T16:17:56.238538+0000","last_peered":"2024-03-04T16:17:56.238538+0000","last_clean":"2024-03-04T16:17:56.238538+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:17:56.238538+0000","last_undegraded":"2024-03-04T16:17:56.238538+0000","last_fullsized":"2024-03-04T16:17:56.238538+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:56.238364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:56.238364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:19.075087+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":624,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490268+0000","last_change":"2024-03-04T16:17:51.289928+0000","last_active":"2024-03-04T16:17:58.490268+0000","last_peered":"2024-03-04T16:17:58.490268+0000","last_clean":"2024-03-04T16:17:58.490268+0000","last_became_active":"2024-03-04T16:10:29.354187+0000","last_became_peered":"2024-03-04T16:10:29.354187+0000","last_unstale":"2024-03-04T16:17:58.490268+0000","last_undegraded":"2024-03-04T16:17:58.490268+0000","last_fullsized":"2024-03-04T16:17:58.490268+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:51.289785+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:17:51.289785+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:10.506806+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":543,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.484261+0000","last_change":"2024-03-04T16:16:00.082788+0000","last_active":"2024-03-04T16:17:58.484261+0000","last_peered":"2024-03-04T16:17:58.484261+0000","last_clean":"2024-03-04T16:17:58.484261+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:17:58.484261+0000","last_undegraded":"2024-03-04T16:17:58.484261+0000","last_fullsized":"2024-03-04T16:17:58.484261+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:00.082581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:16:00.082581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":559,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490104+0000","last_change":"2024-03-04T16:17:54.262357+0000","last_active":"2024-03-04T16:17:58.490104+0000","last_peered":"2024-03-04T16:17:58.490104+0000","last_clean":"2024-03-04T16:17:58.490104+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:17:58.490104+0000","last_undegraded":"2024-03-04T16:17:58.490104+0000","last_fullsized":"2024-03-04T16:17:58.490104+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:54.262228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.262228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:21.638456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":563,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426630+0000","last_change":"2024-03-04T16:16:57.350376+0000","last_active":"2024-03-04T16:17:52.426630+0000","last_peered":"2024-03-04T16:17:52.426630+0000","last_clean":"2024-03-04T16:17:52.426630+0000","last_became_active":"2024-03-04T16:10:29.246771+0000","last_became_peered":"2024-03-04T16:10:29.246771+0000","last_unstale":"2024-03-04T16:17:52.426630+0000","last_undegraded":"2024-03-04T16:17:52.426630+0000","last_fullsized":"2024-03-04T16:17:52.426630+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.350240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.350240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.159288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"240'19","reported_seq":602,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431550+0000","last_change":"2024-03-04T16:15:56.553897+0000","last_active":"2024-03-04T16:17:52.431550+0000","last_peered":"2024-03-04T16:17:52.431550+0000","last_clean":"2024-03-04T16:17:52.431550+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:17:52.431550+0000","last_undegraded":"2024-03-04T16:17:52.431550+0000","last_fullsized":"2024-03-04T16:17:52.431550+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:15:56.553745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.553745+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":426,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424512+0000","last_change":"2024-03-04T16:16:57.407575+0000","last_active":"2024-03-04T16:17:52.424512+0000","last_peered":"2024-03-04T16:17:52.424512+0000","last_clean":"2024-03-04T16:17:52.424512+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:17:52.424512+0000","last_undegraded":"2024-03-04T16:17:52.424512+0000","last_fullsized":"2024-03-04T16:17:52.424512+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.407499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.407499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.214984+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":761,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430799+0000","last_change":"2024-03-04T16:16:59.613637+0000","last_active":"2024-03-04T16:17:52.430799+0000","last_peered":"2024-03-04T16:17:52.430799+0000","last_clean":"2024-03-04T16:17:52.430799+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:17:52.430799+0000","last_undegraded":"2024-03-04T16:17:52.430799+0000","last_fullsized":"2024-03-04T16:17:52.430799+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:16:59.613549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.613549+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:08.905918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":591,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424016+0000","last_change":"2024-03-04T16:16:14.389925+0000","last_active":"2024-03-04T16:17:52.424016+0000","last_peered":"2024-03-04T16:17:52.424016+0000","last_clean":"2024-03-04T16:17:52.424016+0000","last_became_active":"2024-03-04T16:10:29.240732+0000","last_became_peered":"2024-03-04T16:10:29.240732+0000","last_unstale":"2024-03-04T16:17:52.424016+0000","last_undegraded":"2024-03-04T16:17:52.424016+0000","last_fullsized":"2024-03-04T16:17:52.424016+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:14.389807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.389807+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":605,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.494364+0000","last_change":"2024-03-04T16:16:53.775738+0000","last_active":"2024-03-04T16:17:58.494364+0000","last_peered":"2024-03-04T16:17:58.494364+0000","last_clean":"2024-03-04T16:17:58.494364+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:17:58.494364+0000","last_undegraded":"2024-03-04T16:17:58.494364+0000","last_fullsized":"2024-03-04T16:17:58.494364+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:53.775609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.775609+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.927920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":551,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429881+0000","last_change":"2024-03-04T16:17:50.474749+0000","last_active":"2024-03-04T16:17:52.429881+0000","last_peered":"2024-03-04T16:17:52.429881+0000","last_clean":"2024-03-04T16:17:52.429881+0000","last_became_active":"2024-03-04T16:10:17.463758+0000","last_became_peered":"2024-03-04T16:10:17.463758+0000","last_unstale":"2024-03-04T16:17:52.429881+0000","last_undegraded":"2024-03-04T16:17:52.429881+0000","last_fullsized":"2024-03-04T16:17:52.429881+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.474636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.474636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:13.786735+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"192'10","reported_seq":737,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490456+0000","last_change":"2024-03-04T16:15:10.811100+0000","last_active":"2024-03-04T16:17:58.490456+0000","last_peered":"2024-03-04T16:17:58.490456+0000","last_clean":"2024-03-04T16:17:58.490456+0000","last_became_active":"2024-03-04T16:10:17.483274+0000","last_became_peered":"2024-03-04T16:10:17.483274+0000","last_unstale":"2024-03-04T16:17:58.490456+0000","last_undegraded":"2024-03-04T16:17:58.490456+0000","last_fullsized":"2024-03-04T16:17:58.490456+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:15:10.810975+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-04T16:04:31.380175+0000","last_clean_scrub_stamp":"2024-03-04T16:15:10.810975+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":458,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.423822+0000","last_change":"2024-03-04T16:17:47.360887+0000","last_active":"2024-03-04T16:17:52.423822+0000","last_peered":"2024-03-04T16:17:52.423822+0000","last_clean":"2024-03-04T16:17:52.423822+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:17:52.423822+0000","last_undegraded":"2024-03-04T16:17:52.423822+0000","last_fullsized":"2024-03-04T16:17:52.423822+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:17:47.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.360747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:17.198722+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":577,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431057+0000","last_change":"2024-03-04T16:15:07.770440+0000","last_active":"2024-03-04T16:17:52.431057+0000","last_peered":"2024-03-04T16:17:52.431057+0000","last_clean":"2024-03-04T16:17:52.431057+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:17:52.431057+0000","last_undegraded":"2024-03-04T16:17:52.431057+0000","last_fullsized":"2024-03-04T16:17:52.431057+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:15:07.770201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:07.770201+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.427000+0000","last_change":"2024-03-04T16:17:43.332361+0000","last_active":"2024-03-04T16:17:52.427000+0000","last_peered":"2024-03-04T16:17:52.427000+0000","last_clean":"2024-03-04T16:17:52.427000+0000","last_became_active":"2024-03-04T16:10:17.462554+0000","last_became_peered":"2024-03-04T16:10:17.462554+0000","last_unstale":"2024-03-04T16:17:52.427000+0000","last_undegraded":"2024-03-04T16:17:52.427000+0000","last_fullsized":"2024-03-04T16:17:52.427000+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:59.645120+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":543,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491772+0000","last_change":"2024-03-04T16:17:55.208057+0000","last_active":"2024-03-04T16:17:58.491772+0000","last_peered":"2024-03-04T16:17:58.491772+0000","last_clean":"2024-03-04T16:17:58.491772+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:17:58.491772+0000","last_undegraded":"2024-03-04T16:17:58.491772+0000","last_fullsized":"2024-03-04T16:17:58.491772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:55.207977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.207977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:03.102139+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":563,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.432968+0000","last_change":"2024-03-04T16:16:55.383375+0000","last_active":"2024-03-04T16:17:52.432968+0000","last_peered":"2024-03-04T16:17:52.432968+0000","last_clean":"2024-03-04T16:17:52.432968+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:17:52.432968+0000","last_undegraded":"2024-03-04T16:17:52.432968+0000","last_fullsized":"2024-03-04T16:17:52.432968+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.383292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.383292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.382040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":422,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.496993+0000","last_change":"2024-03-04T16:16:53.856464+0000","last_active":"2024-03-04T16:17:58.496993+0000","last_peered":"2024-03-04T16:17:58.496993+0000","last_clean":"2024-03-04T16:17:58.496993+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:17:58.496993+0000","last_undegraded":"2024-03-04T16:17:58.496993+0000","last_fullsized":"2024-03-04T16:17:58.496993+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:16:53.856361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.856361+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.016107+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":559,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429463+0000","last_change":"2024-03-04T16:16:56.475032+0000","last_active":"2024-03-04T16:17:52.429463+0000","last_peered":"2024-03-04T16:17:52.429463+0000","last_clean":"2024-03-04T16:17:52.429463+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:17:52.429463+0000","last_undegraded":"2024-03-04T16:17:52.429463+0000","last_fullsized":"2024-03-04T16:17:52.429463+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:56.474881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:16:56.474881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.070090+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":548,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430932+0000","last_change":"2024-03-04T16:15:56.777453+0000","last_active":"2024-03-04T16:17:52.430932+0000","last_peered":"2024-03-04T16:17:52.430932+0000","last_clean":"2024-03-04T16:17:52.430932+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:17:52.430932+0000","last_undegraded":"2024-03-04T16:17:52.430932+0000","last_fullsized":"2024-03-04T16:17:52.430932+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:56.777347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.777347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":465,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.498552+0000","last_change":"2024-03-04T16:16:51.831340+0000","last_active":"2024-03-04T16:17:58.498552+0000","last_peered":"2024-03-04T16:17:58.498552+0000","last_clean":"2024-03-04T16:17:58.498552+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:17:58.498552+0000","last_undegraded":"2024-03-04T16:17:58.498552+0000","last_fullsized":"2024-03-04T16:17:58.498552+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:51.830496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:16:51.830496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.535280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491978+0000","last_change":"2024-03-04T16:14:53.778826+0000","last_active":"2024-03-04T16:17:58.491978+0000","last_peered":"2024-03-04T16:17:58.491978+0000","last_clean":"2024-03-04T16:17:58.491978+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:17:58.491978+0000","last_undegraded":"2024-03-04T16:17:58.491978+0000","last_fullsized":"2024-03-04T16:17:58.491978+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:14:53.778693+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.778693+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"314'5868","reported_seq":5914,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.945496+0000","last_change":"2024-03-04T16:16:53.660438+0000","last_active":"2024-03-04T16:17:57.945496+0000","last_peered":"2024-03-04T16:17:57.945496+0000","last_clean":"2024-03-04T16:17:57.945496+0000","last_became_active":"2024-03-04T16:16:53.660156+0000","last_became_peered":"2024-03-04T16:16:53.660156+0000","last_unstale":"2024-03-04T16:17:57.945496+0000","last_undegraded":"2024-03-04T16:17:57.945496+0000","last_fullsized":"2024-03-04T16:17:57.945496+0000","mapping_epoch":296,"log_start":"304'1300","ondisk_log_start":"304'1300","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.297306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384565248,"num_objects":5868,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5868,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428064+0000","last_change":"2024-03-04T16:15:48.981473+0000","last_active":"2024-03-04T16:17:52.428064+0000","last_peered":"2024-03-04T16:17:52.428064+0000","last_clean":"2024-03-04T16:17:52.428064+0000","last_became_active":"2024-03-04T16:10:29.340463+0000","last_became_peered":"2024-03-04T16:10:29.340463+0000","last_unstale":"2024-03-04T16:17:52.428064+0000","last_undegraded":"2024-03-04T16:17:52.428064+0000","last_fullsized":"2024-03-04T16:17:52.428064+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:48.981357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.981357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":601,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428353+0000","last_change":"2024-03-04T16:17:07.251362+0000","last_active":"2024-03-04T16:17:52.428353+0000","last_peered":"2024-03-04T16:17:52.428353+0000","last_clean":"2024-03-04T16:17:52.428353+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:17:52.428353+0000","last_undegraded":"2024-03-04T16:17:52.428353+0000","last_fullsized":"2024-03-04T16:17:52.428353+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:17:07.251155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.251155+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.475039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":626,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:54.552434+0000","last_change":"2024-03-04T16:17:54.552434+0000","last_active":"2024-03-04T16:17:54.552434+0000","last_peered":"2024-03-04T16:17:54.552434+0000","last_clean":"2024-03-04T16:17:54.552434+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:17:54.552434+0000","last_undegraded":"2024-03-04T16:17:54.552434+0000","last_fullsized":"2024-03-04T16:17:54.552434+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:17:54.552272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.552272+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:11.668816+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":758,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.436358+0000","last_change":"2024-03-04T16:17:09.268256+0000","last_active":"2024-03-04T16:17:52.436358+0000","last_peered":"2024-03-04T16:17:52.436358+0000","last_clean":"2024-03-04T16:17:52.436358+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:17:52.436358+0000","last_undegraded":"2024-03-04T16:17:52.436358+0000","last_fullsized":"2024-03-04T16:17:52.436358+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:17:09.268137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:09.268137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.542889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"315'5800","reported_seq":5848,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.897994+0000","last_change":"2024-03-04T16:16:53.663167+0000","last_active":"2024-03-04T16:17:58.897994+0000","last_peered":"2024-03-04T16:17:58.897994+0000","last_clean":"2024-03-04T16:17:58.897994+0000","last_became_active":"2024-03-04T16:16:53.661370+0000","last_became_peered":"2024-03-04T16:16:53.661370+0000","last_unstale":"2024-03-04T16:17:58.897994+0000","last_undegraded":"2024-03-04T16:17:58.897994+0000","last_fullsized":"2024-03-04T16:17:58.897994+0000","mapping_epoch":296,"log_start":"304'1700","ondisk_log_start":"304'1700","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4100,"log_dups_size":0,"ondisk_log_size":4100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.342888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380108800,"num_objects":5800,"num_object_clones":0,"num_object_copies":17400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5800,"num_write_kb":371200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":539,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491796+0000","last_change":"2024-03-04T16:16:18.274771+0000","last_active":"2024-03-04T16:17:58.491796+0000","last_peered":"2024-03-04T16:17:58.491796+0000","last_clean":"2024-03-04T16:17:58.491796+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:17:58.491796+0000","last_undegraded":"2024-03-04T16:17:58.491796+0000","last_fullsized":"2024-03-04T16:17:58.491796+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.274659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.274659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":565,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428396+0000","last_change":"2024-03-04T16:16:20.383892+0000","last_active":"2024-03-04T16:17:52.428396+0000","last_peered":"2024-03-04T16:17:52.428396+0000","last_clean":"2024-03-04T16:17:52.428396+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:17:52.428396+0000","last_undegraded":"2024-03-04T16:17:52.428396+0000","last_fullsized":"2024-03-04T16:17:52.428396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.383782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.383782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":586,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428678+0000","last_change":"2024-03-04T16:16:08.552789+0000","last_active":"2024-03-04T16:17:52.428678+0000","last_peered":"2024-03-04T16:17:52.428678+0000","last_clean":"2024-03-04T16:17:52.428678+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:17:52.428678+0000","last_undegraded":"2024-03-04T16:17:52.428678+0000","last_fullsized":"2024-03-04T16:17:52.428678+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:16:08.552635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.552635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.483520+0000","last_change":"2024-03-04T16:16:24.044051+0000","last_active":"2024-03-04T16:17:58.483520+0000","last_peered":"2024-03-04T16:17:58.483520+0000","last_clean":"2024-03-04T16:17:58.483520+0000","last_became_active":"2024-03-04T16:16:24.043673+0000","last_became_peered":"2024-03-04T16:16:24.043673+0000","last_unstale":"2024-03-04T16:17:58.483520+0000","last_undegraded":"2024-03-04T16:17:58.483520+0000","last_fullsized":"2024-03-04T16:17:58.483520+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:46.262241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:46.262241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.4","version":"315'5813","reported_seq":5861,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.528138+0000","last_change":"2024-03-04T16:16:53.660736+0000","last_active":"2024-03-04T16:17:58.528138+0000","last_peered":"2024-03-04T16:17:58.528138+0000","last_clean":"2024-03-04T16:17:58.528138+0000","last_became_active":"2024-03-04T16:16:53.660464+0000","last_became_peered":"2024-03-04T16:16:53.660464+0000","last_unstale":"2024-03-04T16:17:58.528138+0000","last_undegraded":"2024-03-04T16:17:58.528138+0000","last_fullsized":"2024-03-04T16:17:58.528138+0000","mapping_epoch":296,"log_start":"305'1900","ondisk_log_start":"305'1900","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3913,"log_dups_size":0,"ondisk_log_size":3913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:01.967586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":17439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5813,"num_write_kb":372032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":565,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.307671+0000","last_change":"2024-03-04T16:17:57.307709+0000","last_active":"2024-03-04T16:17:57.307671+0000","last_peered":"2024-03-04T16:17:57.307671+0000","last_clean":"2024-03-04T16:17:57.307671+0000","last_became_active":"2024-03-04T16:10:29.248377+0000","last_became_peered":"2024-03-04T16:10:29.248377+0000","last_unstale":"2024-03-04T16:17:57.307671+0000","last_undegraded":"2024-03-04T16:17:57.307671+0000","last_fullsized":"2024-03-04T16:17:57.307671+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:57.307587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:57.307587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:13.177846+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":287,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495832+0000","last_change":"2024-03-04T16:16:19.642828+0000","last_active":"2024-03-04T16:17:58.495832+0000","last_peered":"2024-03-04T16:17:58.495832+0000","last_clean":"2024-03-04T16:17:58.495832+0000","last_became_active":"2024-03-04T16:10:17.466170+0000","last_became_peered":"2024-03-04T16:10:17.466170+0000","last_unstale":"2024-03-04T16:17:58.495832+0000","last_undegraded":"2024-03-04T16:17:58.495832+0000","last_fullsized":"2024-03-04T16:17:58.495832+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.642722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.642722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":571,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429787+0000","last_change":"2024-03-04T16:17:46.310631+0000","last_active":"2024-03-04T16:17:52.429787+0000","last_peered":"2024-03-04T16:17:52.429787+0000","last_clean":"2024-03-04T16:17:52.429787+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:17:52.429787+0000","last_undegraded":"2024-03-04T16:17:52.429787+0000","last_fullsized":"2024-03-04T16:17:52.429787+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.310358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.310358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:15.200945+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":374,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489858+0000","last_change":"2024-03-04T16:17:58.315576+0000","last_active":"2024-03-04T16:17:58.489858+0000","last_peered":"2024-03-04T16:17:58.489858+0000","last_clean":"2024-03-04T16:17:58.489858+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:17:58.489858+0000","last_undegraded":"2024-03-04T16:17:58.489858+0000","last_fullsized":"2024-03-04T16:17:58.489858+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:17:58.315405+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.315405+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:10.838870+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"315'5901","reported_seq":5949,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.517936+0000","last_change":"2024-03-04T16:16:53.660460+0000","last_active":"2024-03-04T16:17:58.517936+0000","last_peered":"2024-03-04T16:17:58.517936+0000","last_clean":"2024-03-04T16:17:58.517936+0000","last_became_active":"2024-03-04T16:16:53.660170+0000","last_became_peered":"2024-03-04T16:16:53.660170+0000","last_unstale":"2024-03-04T16:17:58.517936+0000","last_undegraded":"2024-03-04T16:17:58.517936+0000","last_fullsized":"2024-03-04T16:17:58.517936+0000","mapping_epoch":296,"log_start":"305'2000","ondisk_log_start":"305'2000","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.653120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386727936,"num_objects":5901,"num_object_clones":0,"num_object_copies":17703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5901,"num_write_kb":377664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":433,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.432723+0000","last_change":"2024-03-04T16:16:12.401505+0000","last_active":"2024-03-04T16:17:52.432723+0000","last_peered":"2024-03-04T16:17:52.432723+0000","last_clean":"2024-03-04T16:17:52.432723+0000","last_became_active":"2024-03-04T16:10:17.462155+0000","last_became_peered":"2024-03-04T16:10:17.462155+0000","last_unstale":"2024-03-04T16:17:52.432723+0000","last_undegraded":"2024-03-04T16:17:52.432723+0000","last_fullsized":"2024-03-04T16:17:52.432723+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.401385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426525+0000","last_change":"2024-03-04T16:17:52.337255+0000","last_active":"2024-03-04T16:17:52.426525+0000","last_peered":"2024-03-04T16:17:52.426525+0000","last_clean":"2024-03-04T16:17:52.426525+0000","last_became_active":"2024-03-04T16:10:17.461948+0000","last_became_peered":"2024-03-04T16:10:17.461948+0000","last_unstale":"2024-03-04T16:17:52.426525+0000","last_undegraded":"2024-03-04T16:17:52.426525+0000","last_fullsized":"2024-03-04T16:17:52.426525+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:52.337129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:17:52.337129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:16.036400+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":562,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430348+0000","last_change":"2024-03-04T16:15:53.841693+0000","last_active":"2024-03-04T16:17:52.430348+0000","last_peered":"2024-03-04T16:17:52.430348+0000","last_clean":"2024-03-04T16:17:52.430348+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:17:52.430348+0000","last_undegraded":"2024-03-04T16:17:52.430348+0000","last_fullsized":"2024-03-04T16:17:52.430348+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:53.841578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:15:53.841578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":422,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495693+0000","last_change":"2024-03-04T16:16:49.762334+0000","last_active":"2024-03-04T16:17:58.495693+0000","last_peered":"2024-03-04T16:17:58.495693+0000","last_clean":"2024-03-04T16:17:58.495693+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:17:58.495693+0000","last_undegraded":"2024-03-04T16:17:58.495693+0000","last_fullsized":"2024-03-04T16:17:58.495693+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:16:49.762239+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.762239+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.560260+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"315'5793","reported_seq":5841,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.540193+0000","last_change":"2024-03-04T16:16:53.660429+0000","last_active":"2024-03-04T16:17:58.540193+0000","last_peered":"2024-03-04T16:17:58.540193+0000","last_clean":"2024-03-04T16:17:58.540193+0000","last_became_active":"2024-03-04T16:16:53.660115+0000","last_became_peered":"2024-03-04T16:16:53.660115+0000","last_unstale":"2024-03-04T16:17:58.540193+0000","last_undegraded":"2024-03-04T16:17:58.540193+0000","last_fullsized":"2024-03-04T16:17:58.540193+0000","mapping_epoch":296,"log_start":"305'1800","ondisk_log_start":"305'1800","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.638211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379650048,"num_objects":5793,"num_object_clones":0,"num_object_copies":17379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":370752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":532,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489148+0000","last_change":"2024-03-04T16:14:51.811044+0000","last_active":"2024-03-04T16:17:58.489148+0000","last_peered":"2024-03-04T16:17:58.489148+0000","last_clean":"2024-03-04T16:17:58.489148+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:17:58.489148+0000","last_undegraded":"2024-03-04T16:17:58.489148+0000","last_fullsized":"2024-03-04T16:17:58.489148+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:51.810913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:14:51.810913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":578,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.427909+0000","last_change":"2024-03-04T16:15:51.894739+0000","last_active":"2024-03-04T16:17:52.427909+0000","last_peered":"2024-03-04T16:17:52.427909+0000","last_clean":"2024-03-04T16:17:52.427909+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:17:52.427909+0000","last_undegraded":"2024-03-04T16:17:52.427909+0000","last_fullsized":"2024-03-04T16:17:52.427909+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:15:51.894452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.894452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":576,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431356+0000","last_change":"2024-03-04T16:15:45.509761+0000","last_active":"2024-03-04T16:17:52.431356+0000","last_peered":"2024-03-04T16:17:52.431356+0000","last_clean":"2024-03-04T16:17:52.431356+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:17:52.431356+0000","last_undegraded":"2024-03-04T16:17:52.431356+0000","last_fullsized":"2024-03-04T16:17:52.431356+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:45.509639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:15:45.509639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"240'42","reported_seq":861,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491462+0000","last_change":"2024-03-04T16:16:17.407977+0000","last_active":"2024-03-04T16:17:58.491462+0000","last_peered":"2024-03-04T16:17:58.491462+0000","last_clean":"2024-03-04T16:17:58.491462+0000","last_became_active":"2024-03-04T16:10:29.252340+0000","last_became_peered":"2024-03-04T16:10:29.252340+0000","last_unstale":"2024-03-04T16:17:58.491462+0000","last_undegraded":"2024-03-04T16:17:58.491462+0000","last_fullsized":"2024-03-04T16:17:58.491462+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:16:17.407775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.407775+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":403,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.498376+0000","last_change":"2024-03-04T16:17:01.832332+0000","last_active":"2024-03-04T16:17:58.498376+0000","last_peered":"2024-03-04T16:17:58.498376+0000","last_clean":"2024-03-04T16:17:58.498376+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:17:58.498376+0000","last_undegraded":"2024-03-04T16:17:58.498376+0000","last_fullsized":"2024-03-04T16:17:58.498376+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.832115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.832115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:27.727695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"314'5860","reported_seq":5906,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.979709+0000","last_change":"2024-03-04T16:16:53.657683+0000","last_active":"2024-03-04T16:17:57.979709+0000","last_peered":"2024-03-04T16:17:57.979709+0000","last_clean":"2024-03-04T16:17:57.979709+0000","last_became_active":"2024-03-04T16:16:53.657368+0000","last_became_peered":"2024-03-04T16:16:53.657368+0000","last_unstale":"2024-03-04T16:17:57.979709+0000","last_undegraded":"2024-03-04T16:17:57.979709+0000","last_fullsized":"2024-03-04T16:17:57.979709+0000","mapping_epoch":296,"log_start":"304'1300","ondisk_log_start":"304'1300","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384040960,"num_objects":5860,"num_object_clones":0,"num_object_copies":17580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5860,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5860,"num_write_kb":375040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":560,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429924+0000","last_change":"2024-03-04T16:17:50.390361+0000","last_active":"2024-03-04T16:17:52.429924+0000","last_peered":"2024-03-04T16:17:52.429924+0000","last_clean":"2024-03-04T16:17:52.429924+0000","last_became_active":"2024-03-04T16:10:29.371008+0000","last_became_peered":"2024-03-04T16:10:29.371008+0000","last_unstale":"2024-03-04T16:17:52.429924+0000","last_undegraded":"2024-03-04T16:17:52.429924+0000","last_fullsized":"2024-03-04T16:17:52.429924+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.390204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.390204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:53.317685+0000","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":572,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489209+0000","last_change":"2024-03-04T16:16:14.435854+0000","last_active":"2024-03-04T16:17:58.489209+0000","last_peered":"2024-03-04T16:17:58.489209+0000","last_clean":"2024-03-04T16:17:58.489209+0000","last_became_active":"2024-03-04T16:10:29.351733+0000","last_became_peered":"2024-03-04T16:10:29.351733+0000","last_unstale":"2024-03-04T16:17:58.489209+0000","last_undegraded":"2024-03-04T16:17:58.489209+0000","last_fullsized":"2024-03-04T16:17:58.489209+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:14.435737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:14.435737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":280,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.484358+0000","last_change":"2024-03-04T16:16:10.021523+0000","last_active":"2024-03-04T16:17:58.484358+0000","last_peered":"2024-03-04T16:17:58.484358+0000","last_clean":"2024-03-04T16:17:58.484358+0000","last_became_active":"2024-03-04T16:10:17.479874+0000","last_became_peered":"2024-03-04T16:10:17.479874+0000","last_unstale":"2024-03-04T16:17:58.484358+0000","last_undegraded":"2024-03-04T16:17:58.484358+0000","last_fullsized":"2024-03-04T16:17:58.484358+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.021439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":203,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490711+0000","last_change":"2024-03-04T16:16:57.389891+0000","last_active":"2024-03-04T16:17:58.490711+0000","last_peered":"2024-03-04T16:17:58.490711+0000","last_clean":"2024-03-04T16:17:58.490711+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:17:58.490711+0000","last_undegraded":"2024-03-04T16:17:58.490711+0000","last_fullsized":"2024-03-04T16:17:58.490711+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:16:57.389758+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.389758+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:18.714260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":277,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431105+0000","last_change":"2024-03-04T16:16:19.696457+0000","last_active":"2024-03-04T16:17:52.431105+0000","last_peered":"2024-03-04T16:17:52.431105+0000","last_clean":"2024-03-04T16:17:52.431105+0000","last_became_active":"2024-03-04T16:10:17.466014+0000","last_became_peered":"2024-03-04T16:10:17.466014+0000","last_unstale":"2024-03-04T16:17:52.431105+0000","last_undegraded":"2024-03-04T16:17:52.431105+0000","last_fullsized":"2024-03-04T16:17:52.431105+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.696369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.696369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.b","version":"315'5857","reported_seq":5905,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.684081+0000","last_change":"2024-03-04T16:16:53.666174+0000","last_active":"2024-03-04T16:17:58.684081+0000","last_peered":"2024-03-04T16:17:58.684081+0000","last_clean":"2024-03-04T16:17:58.684081+0000","last_became_active":"2024-03-04T16:16:53.665651+0000","last_became_peered":"2024-03-04T16:16:53.665651+0000","last_unstale":"2024-03-04T16:17:58.684081+0000","last_undegraded":"2024-03-04T16:17:58.684081+0000","last_fullsized":"2024-03-04T16:17:58.684081+0000","mapping_epoch":296,"log_start":"305'2000","ondisk_log_start":"305'2000","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3857,"log_dups_size":0,"ondisk_log_size":3857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.126117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383844352,"num_objects":5857,"num_object_clones":0,"num_object_copies":17571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5857,"num_write_kb":374848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":552,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430679+0000","last_change":"2024-03-04T16:17:46.756768+0000","last_active":"2024-03-04T16:17:52.430679+0000","last_peered":"2024-03-04T16:17:52.430679+0000","last_clean":"2024-03-04T16:17:52.430679+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:17:52.430679+0000","last_undegraded":"2024-03-04T16:17:52.430679+0000","last_fullsized":"2024-03-04T16:17:52.430679+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.756659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.756659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:58.209587+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":865,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424767+0000","last_change":"2024-03-04T16:16:01.374602+0000","last_active":"2024-03-04T16:17:52.424767+0000","last_peered":"2024-03-04T16:17:52.424767+0000","last_clean":"2024-03-04T16:17:52.424767+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:17:52.424767+0000","last_undegraded":"2024-03-04T16:17:52.424767+0000","last_fullsized":"2024-03-04T16:17:52.424767+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:16:01.374477+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.374477+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":558,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489431+0000","last_change":"2024-03-04T16:16:20.104755+0000","last_active":"2024-03-04T16:17:58.489431+0000","last_peered":"2024-03-04T16:17:58.489431+0000","last_clean":"2024-03-04T16:17:58.489431+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:17:58.489431+0000","last_undegraded":"2024-03-04T16:17:58.489431+0000","last_fullsized":"2024-03-04T16:17:58.489431+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.104643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.104643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":438,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426517+0000","last_change":"2024-03-04T16:17:05.303664+0000","last_active":"2024-03-04T16:17:52.426517+0000","last_peered":"2024-03-04T16:17:52.426517+0000","last_clean":"2024-03-04T16:17:52.426517+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:17:52.426517+0000","last_undegraded":"2024-03-04T16:17:52.426517+0000","last_fullsized":"2024-03-04T16:17:52.426517+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:17:05.303531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.303531+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:29.512468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":564,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428926+0000","last_change":"2024-03-04T16:16:15.349131+0000","last_active":"2024-03-04T16:17:52.428926+0000","last_peered":"2024-03-04T16:17:52.428926+0000","last_clean":"2024-03-04T16:17:52.428926+0000","last_became_active":"2024-03-04T16:10:17.465329+0000","last_became_peered":"2024-03-04T16:10:17.465329+0000","last_unstale":"2024-03-04T16:17:52.428926+0000","last_undegraded":"2024-03-04T16:17:52.428926+0000","last_fullsized":"2024-03-04T16:17:52.428926+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:15.348942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:15.348942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"314'5954","reported_seq":6000,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.690247+0000","last_change":"2024-03-04T16:16:53.660223+0000","last_active":"2024-03-04T16:17:57.690247+0000","last_peered":"2024-03-04T16:17:57.690247+0000","last_clean":"2024-03-04T16:17:57.690247+0000","last_became_active":"2024-03-04T16:16:53.659965+0000","last_became_peered":"2024-03-04T16:16:53.659965+0000","last_unstale":"2024-03-04T16:17:57.690247+0000","last_undegraded":"2024-03-04T16:17:57.690247+0000","last_fullsized":"2024-03-04T16:17:57.690247+0000","mapping_epoch":296,"log_start":"304'1600","ondisk_log_start":"304'1600","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:17:59.464393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390201344,"num_objects":5954,"num_object_clones":0,"num_object_copies":17862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5954,"num_write_kb":381056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":556,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489761+0000","last_change":"2024-03-04T16:16:55.443627+0000","last_active":"2024-03-04T16:17:58.489761+0000","last_peered":"2024-03-04T16:17:58.489761+0000","last_clean":"2024-03-04T16:17:58.489761+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:17:58.489761+0000","last_undegraded":"2024-03-04T16:17:58.489761+0000","last_fullsized":"2024-03-04T16:17:58.489761+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.443547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.443547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.590784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":411,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.496744+0000","last_change":"2024-03-04T16:17:02.856208+0000","last_active":"2024-03-04T16:17:58.496744+0000","last_peered":"2024-03-04T16:17:58.496744+0000","last_clean":"2024-03-04T16:17:58.496744+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:17:58.496744+0000","last_undegraded":"2024-03-04T16:17:58.496744+0000","last_fullsized":"2024-03-04T16:17:58.496744+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:02.856124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:02.856124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:04.756539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1486,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:56.354317+0000","last_change":"2024-03-04T16:16:10.322558+0000","last_active":"2024-03-04T16:17:56.354317+0000","last_peered":"2024-03-04T16:17:56.354317+0000","last_clean":"2024-03-04T16:17:56.354317+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:17:56.354317+0000","last_undegraded":"2024-03-04T16:17:56.354317+0000","last_fullsized":"2024-03-04T16:17:56.354317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:16:10.322412+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.322412+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":907,"num_read_kb":902,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":455,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495729+0000","last_change":"2024-03-04T16:17:53.588697+0000","last_active":"2024-03-04T16:17:58.495729+0000","last_peered":"2024-03-04T16:17:58.495729+0000","last_clean":"2024-03-04T16:17:58.495729+0000","last_became_active":"2024-03-04T16:10:17.477789+0000","last_became_peered":"2024-03-04T16:10:17.477789+0000","last_unstale":"2024-03-04T16:17:58.495729+0000","last_undegraded":"2024-03-04T16:17:58.495729+0000","last_fullsized":"2024-03-04T16:17:58.495729+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:53.588494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.588494+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:55.569754+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":564,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431057+0000","last_change":"2024-03-04T16:17:43.601364+0000","last_active":"2024-03-04T16:17:52.431057+0000","last_peered":"2024-03-04T16:17:52.431057+0000","last_clean":"2024-03-04T16:17:52.431057+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:17:52.431057+0000","last_undegraded":"2024-03-04T16:17:52.431057+0000","last_fullsized":"2024-03-04T16:17:52.431057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.600397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.600397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:04.934225+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"314'5714","reported_seq":5760,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.706739+0000","last_change":"2024-03-04T16:16:53.661446+0000","last_active":"2024-03-04T16:17:57.706739+0000","last_peered":"2024-03-04T16:17:57.706739+0000","last_clean":"2024-03-04T16:17:57.706739+0000","last_became_active":"2024-03-04T16:16:53.661113+0000","last_became_peered":"2024-03-04T16:16:53.661113+0000","last_unstale":"2024-03-04T16:17:57.706739+0000","last_undegraded":"2024-03-04T16:17:57.706739+0000","last_fullsized":"2024-03-04T16:17:57.706739+0000","mapping_epoch":296,"log_start":"305'1800","ondisk_log_start":"305'1800","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3914,"log_dups_size":0,"ondisk_log_size":3914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.556881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374472704,"num_objects":5714,"num_object_clones":0,"num_object_copies":17142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5714,"num_write_kb":365696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489360+0000","last_change":"2024-03-04T16:16:11.401122+0000","last_active":"2024-03-04T16:17:58.489360+0000","last_peered":"2024-03-04T16:17:58.489360+0000","last_clean":"2024-03-04T16:17:58.489360+0000","last_became_active":"2024-03-04T16:10:29.247179+0000","last_became_peered":"2024-03-04T16:10:29.247179+0000","last_unstale":"2024-03-04T16:17:58.489360+0000","last_undegraded":"2024-03-04T16:17:58.489360+0000","last_fullsized":"2024-03-04T16:17:58.489360+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.401004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.401004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":558,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491311+0000","last_change":"2024-03-04T16:16:04.339942+0000","last_active":"2024-03-04T16:17:58.491311+0000","last_peered":"2024-03-04T16:17:58.491311+0000","last_clean":"2024-03-04T16:17:58.491311+0000","last_became_active":"2024-03-04T16:10:17.491057+0000","last_became_peered":"2024-03-04T16:10:17.491057+0000","last_unstale":"2024-03-04T16:17:58.491311+0000","last_undegraded":"2024-03-04T16:17:58.491311+0000","last_fullsized":"2024-03-04T16:17:58.491311+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:04.339827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:04.339827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":532,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430424+0000","last_change":"2024-03-04T16:14:53.705612+0000","last_active":"2024-03-04T16:17:52.430424+0000","last_peered":"2024-03-04T16:17:52.430424+0000","last_clean":"2024-03-04T16:17:52.430424+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:17:52.430424+0000","last_undegraded":"2024-03-04T16:17:52.430424+0000","last_fullsized":"2024-03-04T16:17:52.430424+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:14:53.705474+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:14:53.705474+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.427959+0000","last_change":"2024-03-04T16:16:54.600353+0000","last_active":"2024-03-04T16:17:52.427959+0000","last_peered":"2024-03-04T16:17:52.427959+0000","last_clean":"2024-03-04T16:17:52.427959+0000","last_became_active":"2024-03-04T16:10:29.328328+0000","last_became_peered":"2024-03-04T16:10:29.328328+0000","last_unstale":"2024-03-04T16:17:52.427959+0000","last_undegraded":"2024-03-04T16:17:52.427959+0000","last_fullsized":"2024-03-04T16:17:52.427959+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:54.600256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:54.600256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:06.964184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"314'5669","reported_seq":5715,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.689774+0000","last_change":"2024-03-04T16:16:53.660377+0000","last_active":"2024-03-04T16:17:57.689774+0000","last_peered":"2024-03-04T16:17:57.689774+0000","last_clean":"2024-03-04T16:17:57.689774+0000","last_became_active":"2024-03-04T16:16:53.659725+0000","last_became_peered":"2024-03-04T16:16:53.659725+0000","last_unstale":"2024-03-04T16:17:57.689774+0000","last_undegraded":"2024-03-04T16:17:57.689774+0000","last_fullsized":"2024-03-04T16:17:57.689774+0000","mapping_epoch":296,"log_start":"304'1300","ondisk_log_start":"304'1300","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:22.365457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371523584,"num_objects":5669,"num_object_clones":0,"num_object_copies":17007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5669,"num_write_kb":362816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":562,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.497819+0000","last_change":"2024-03-04T16:15:52.388804+0000","last_active":"2024-03-04T16:17:58.497819+0000","last_peered":"2024-03-04T16:17:58.497819+0000","last_clean":"2024-03-04T16:17:58.497819+0000","last_became_active":"2024-03-04T16:10:17.467089+0000","last_became_peered":"2024-03-04T16:10:17.467089+0000","last_unstale":"2024-03-04T16:17:58.497819+0000","last_undegraded":"2024-03-04T16:17:58.497819+0000","last_fullsized":"2024-03-04T16:17:58.497819+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:52.388669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:52.388669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":569,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430856+0000","last_change":"2024-03-04T16:16:01.874625+0000","last_active":"2024-03-04T16:17:52.430856+0000","last_peered":"2024-03-04T16:17:52.430856+0000","last_clean":"2024-03-04T16:17:52.430856+0000","last_became_active":"2024-03-04T16:10:29.367976+0000","last_became_peered":"2024-03-04T16:10:29.367976+0000","last_unstale":"2024-03-04T16:17:52.430856+0000","last_undegraded":"2024-03-04T16:17:52.430856+0000","last_fullsized":"2024-03-04T16:17:52.430856+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:01.874356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.874356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":599,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490192+0000","last_change":"2024-03-04T16:17:55.230783+0000","last_active":"2024-03-04T16:17:58.490192+0000","last_peered":"2024-03-04T16:17:58.490192+0000","last_clean":"2024-03-04T16:17:58.490192+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:17:58.490192+0000","last_undegraded":"2024-03-04T16:17:58.490192+0000","last_fullsized":"2024-03-04T16:17:58.490192+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:17:55.230654+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.230654+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:03.244670+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":388,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430785+0000","last_change":"2024-03-04T16:16:11.811999+0000","last_active":"2024-03-04T16:17:52.430785+0000","last_peered":"2024-03-04T16:17:52.430785+0000","last_clean":"2024-03-04T16:17:52.430785+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:17:52.430785+0000","last_undegraded":"2024-03-04T16:17:52.430785+0000","last_fullsized":"2024-03-04T16:17:52.430785+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:16:11.811878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.811878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"315'5876","reported_seq":5924,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.898390+0000","last_change":"2024-03-04T16:16:53.663518+0000","last_active":"2024-03-04T16:17:58.898390+0000","last_peered":"2024-03-04T16:17:58.898390+0000","last_clean":"2024-03-04T16:17:58.898390+0000","last_became_active":"2024-03-04T16:16:53.662075+0000","last_became_peered":"2024-03-04T16:16:53.662075+0000","last_unstale":"2024-03-04T16:17:58.898390+0000","last_undegraded":"2024-03-04T16:17:58.898390+0000","last_fullsized":"2024-03-04T16:17:58.898390+0000","mapping_epoch":296,"log_start":"304'1800","ondisk_log_start":"304'1800","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4076,"log_dups_size":0,"ondisk_log_size":4076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:19.663744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385089536,"num_objects":5876,"num_object_clones":0,"num_object_copies":17628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5876,"num_write_kb":376064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":566,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.424452+0000","last_change":"2024-03-04T16:16:50.462234+0000","last_active":"2024-03-04T16:17:52.424452+0000","last_peered":"2024-03-04T16:17:52.424452+0000","last_clean":"2024-03-04T16:17:52.424452+0000","last_became_active":"2024-03-04T16:10:29.244562+0000","last_became_peered":"2024-03-04T16:10:29.244562+0000","last_unstale":"2024-03-04T16:17:52.424452+0000","last_undegraded":"2024-03-04T16:17:52.424452+0000","last_fullsized":"2024-03-04T16:17:52.424452+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:50.462116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:16:50.462116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:13.011998+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":407,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.495906+0000","last_change":"2024-03-04T16:16:07.544810+0000","last_active":"2024-03-04T16:17:58.495906+0000","last_peered":"2024-03-04T16:17:58.495906+0000","last_clean":"2024-03-04T16:17:58.495906+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:17:58.495906+0000","last_undegraded":"2024-03-04T16:17:58.495906+0000","last_fullsized":"2024-03-04T16:17:58.495906+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:07.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.544679+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":617,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428492+0000","last_change":"2024-03-04T16:15:48.541576+0000","last_active":"2024-03-04T16:17:52.428492+0000","last_peered":"2024-03-04T16:17:52.428492+0000","last_clean":"2024-03-04T16:17:52.428492+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:17:52.428492+0000","last_undegraded":"2024-03-04T16:17:52.428492+0000","last_fullsized":"2024-03-04T16:17:52.428492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'35","last_scrub_stamp":"2024-03-04T16:15:48.541142+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:15:48.541142+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":735,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489093+0000","last_change":"2024-03-04T16:17:06.654928+0000","last_active":"2024-03-04T16:17:58.489093+0000","last_peered":"2024-03-04T16:17:58.489093+0000","last_clean":"2024-03-04T16:17:58.489093+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:17:58.489093+0000","last_undegraded":"2024-03-04T16:17:58.489093+0000","last_fullsized":"2024-03-04T16:17:58.489093+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:06.654784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.654784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:21.453080+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"314'5823","reported_seq":5869,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.701083+0000","last_change":"2024-03-04T16:16:53.659484+0000","last_active":"2024-03-04T16:17:57.701083+0000","last_peered":"2024-03-04T16:17:57.701083+0000","last_clean":"2024-03-04T16:17:57.701083+0000","last_became_active":"2024-03-04T16:16:53.659122+0000","last_became_peered":"2024-03-04T16:16:53.659122+0000","last_unstale":"2024-03-04T16:17:57.701083+0000","last_undegraded":"2024-03-04T16:17:57.701083+0000","last_fullsized":"2024-03-04T16:17:57.701083+0000","mapping_epoch":296,"log_start":"305'1900","ondisk_log_start":"305'1900","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.313098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381616128,"num_objects":5823,"num_object_clones":0,"num_object_copies":17469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.425983+0000","last_change":"2024-03-04T16:16:11.572661+0000","last_active":"2024-03-04T16:17:52.425983+0000","last_peered":"2024-03-04T16:17:52.425983+0000","last_clean":"2024-03-04T16:17:52.425983+0000","last_became_active":"2024-03-04T16:10:29.246576+0000","last_became_peered":"2024-03-04T16:10:29.246576+0000","last_unstale":"2024-03-04T16:17:52.425983+0000","last_undegraded":"2024-03-04T16:17:52.425983+0000","last_fullsized":"2024-03-04T16:17:52.425983+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:11.572528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.572528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":564,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.492084+0000","last_change":"2024-03-04T16:17:47.139524+0000","last_active":"2024-03-04T16:17:58.492084+0000","last_peered":"2024-03-04T16:17:58.492084+0000","last_clean":"2024-03-04T16:17:58.492084+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:17:58.492084+0000","last_undegraded":"2024-03-04T16:17:58.492084+0000","last_fullsized":"2024-03-04T16:17:58.492084+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.139403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.139403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:08.722405+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":646,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.484374+0000","last_change":"2024-03-04T16:15:03.777164+0000","last_active":"2024-03-04T16:17:58.484374+0000","last_peered":"2024-03-04T16:17:58.484374+0000","last_clean":"2024-03-04T16:17:58.484374+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:17:58.484374+0000","last_undegraded":"2024-03-04T16:17:58.484374+0000","last_fullsized":"2024-03-04T16:17:58.484374+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-03-04T16:15:03.776866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:03.776866+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":550,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428234+0000","last_change":"2024-03-04T16:17:45.302896+0000","last_active":"2024-03-04T16:17:52.428234+0000","last_peered":"2024-03-04T16:17:52.428234+0000","last_clean":"2024-03-04T16:17:52.428234+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:17:52.428234+0000","last_undegraded":"2024-03-04T16:17:52.428234+0000","last_fullsized":"2024-03-04T16:17:52.428234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:45.302787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:45.302787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:02.436489+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"314'5746","reported_seq":5792,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.729837+0000","last_change":"2024-03-04T16:16:53.660602+0000","last_active":"2024-03-04T16:17:57.729837+0000","last_peered":"2024-03-04T16:17:57.729837+0000","last_clean":"2024-03-04T16:17:57.729837+0000","last_became_active":"2024-03-04T16:16:53.660328+0000","last_became_peered":"2024-03-04T16:16:53.660328+0000","last_unstale":"2024-03-04T16:17:57.729837+0000","last_undegraded":"2024-03-04T16:17:57.729837+0000","last_fullsized":"2024-03-04T16:17:57.729837+0000","mapping_epoch":296,"log_start":"304'1800","ondisk_log_start":"304'1800","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3946,"log_dups_size":0,"ondisk_log_size":3946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:15.850706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376569856,"num_objects":5746,"num_object_clones":0,"num_object_copies":17238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5746,"num_write_kb":367744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:53.653208+0000","last_change":"2024-03-04T16:17:53.653243+0000","last_active":"2024-03-04T16:17:53.653208+0000","last_peered":"2024-03-04T16:17:53.653208+0000","last_clean":"2024-03-04T16:17:53.653208+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:17:53.653208+0000","last_undegraded":"2024-03-04T16:17:53.653208+0000","last_fullsized":"2024-03-04T16:17:53.653208+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:53.653131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.653131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:06.844794+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":732,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431074+0000","last_change":"2024-03-04T16:14:59.729903+0000","last_active":"2024-03-04T16:17:52.431074+0000","last_peered":"2024-03-04T16:17:52.431074+0000","last_clean":"2024-03-04T16:17:52.431074+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:17:52.431074+0000","last_undegraded":"2024-03-04T16:17:52.431074+0000","last_fullsized":"2024-03-04T16:17:52.431074+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:14:59.729380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.729380+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":560,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.484324+0000","last_change":"2024-03-04T16:17:44.202256+0000","last_active":"2024-03-04T16:17:58.484324+0000","last_peered":"2024-03-04T16:17:58.484324+0000","last_clean":"2024-03-04T16:17:58.484324+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:17:58.484324+0000","last_undegraded":"2024-03-04T16:17:58.484324+0000","last_fullsized":"2024-03-04T16:17:58.484324+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:44.202137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.202137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:07.661459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"301'34","reported_seq":605,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491215+0000","last_change":"2024-03-04T16:17:05.452594+0000","last_active":"2024-03-04T16:17:58.491215+0000","last_peered":"2024-03-04T16:17:58.491215+0000","last_clean":"2024-03-04T16:17:58.491215+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:17:58.491215+0000","last_undegraded":"2024-03-04T16:17:58.491215+0000","last_fullsized":"2024-03-04T16:17:58.491215+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'34","last_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.452417+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:10.844380+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"315'5816","reported_seq":5864,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.690795+0000","last_change":"2024-03-04T16:16:53.658520+0000","last_active":"2024-03-04T16:17:58.690795+0000","last_peered":"2024-03-04T16:17:58.690795+0000","last_clean":"2024-03-04T16:17:58.690795+0000","last_became_active":"2024-03-04T16:16:53.658240+0000","last_became_peered":"2024-03-04T16:16:53.658240+0000","last_unstale":"2024-03-04T16:17:58.690795+0000","last_undegraded":"2024-03-04T16:17:58.690795+0000","last_fullsized":"2024-03-04T16:17:58.690795+0000","mapping_epoch":296,"log_start":"305'2000","ondisk_log_start":"305'2000","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3816,"log_dups_size":0,"ondisk_log_size":3816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:09.724216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":372224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":542,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.489970+0000","last_change":"2024-03-04T16:15:41.965920+0000","last_active":"2024-03-04T16:17:58.489970+0000","last_peered":"2024-03-04T16:17:58.489970+0000","last_clean":"2024-03-04T16:17:58.489970+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:17:58.489970+0000","last_undegraded":"2024-03-04T16:17:58.489970+0000","last_fullsized":"2024-03-04T16:17:58.489970+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:15:41.965768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:41.965768+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":554,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430971+0000","last_change":"2024-03-04T16:15:58.806390+0000","last_active":"2024-03-04T16:17:52.430971+0000","last_peered":"2024-03-04T16:17:52.430971+0000","last_clean":"2024-03-04T16:17:52.430971+0000","last_became_active":"2024-03-04T16:10:29.342080+0000","last_became_peered":"2024-03-04T16:10:29.342080+0000","last_unstale":"2024-03-04T16:17:52.430971+0000","last_undegraded":"2024-03-04T16:17:52.430971+0000","last_fullsized":"2024-03-04T16:17:52.430971+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.806254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.806254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":556,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.428344+0000","last_change":"2024-03-04T16:16:12.534941+0000","last_active":"2024-03-04T16:17:52.428344+0000","last_peered":"2024-03-04T16:17:52.428344+0000","last_clean":"2024-03-04T16:17:52.428344+0000","last_became_active":"2024-03-04T16:10:17.472074+0000","last_became_peered":"2024-03-04T16:10:17.472074+0000","last_unstale":"2024-03-04T16:17:52.428344+0000","last_undegraded":"2024-03-04T16:17:52.428344+0000","last_fullsized":"2024-03-04T16:17:52.428344+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.534849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.534849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":607,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.430877+0000","last_change":"2024-03-04T16:17:00.348913+0000","last_active":"2024-03-04T16:17:52.430877+0000","last_peered":"2024-03-04T16:17:52.430877+0000","last_clean":"2024-03-04T16:17:52.430877+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:17:52.430877+0000","last_undegraded":"2024-03-04T16:17:52.430877+0000","last_fullsized":"2024-03-04T16:17:52.430877+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:00.348771+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.348771+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:17.680455+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"314'5792","reported_seq":5837,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:57.980603+0000","last_change":"2024-03-04T16:16:53.658266+0000","last_active":"2024-03-04T16:17:57.980603+0000","last_peered":"2024-03-04T16:17:57.980603+0000","last_clean":"2024-03-04T16:17:57.980603+0000","last_became_active":"2024-03-04T16:16:53.657934+0000","last_became_peered":"2024-03-04T16:16:53.657934+0000","last_unstale":"2024-03-04T16:17:57.980603+0000","last_undegraded":"2024-03-04T16:17:57.980603+0000","last_fullsized":"2024-03-04T16:17:57.980603+0000","mapping_epoch":296,"log_start":"304'1200","ondisk_log_start":"304'1200","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4592,"log_dups_size":0,"ondisk_log_size":4592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:03.803374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5792,"num_write_kb":370688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":285,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490488+0000","last_change":"2024-03-04T16:16:16.301977+0000","last_active":"2024-03-04T16:17:58.490488+0000","last_peered":"2024-03-04T16:17:58.490488+0000","last_clean":"2024-03-04T16:17:58.490488+0000","last_became_active":"2024-03-04T16:10:29.323523+0000","last_became_peered":"2024-03-04T16:10:29.323523+0000","last_unstale":"2024-03-04T16:17:58.490488+0000","last_undegraded":"2024-03-04T16:17:58.490488+0000","last_fullsized":"2024-03-04T16:17:58.490488+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.301767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.301767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":551,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.491563+0000","last_change":"2024-03-04T16:16:07.370204+0000","last_active":"2024-03-04T16:17:58.491563+0000","last_peered":"2024-03-04T16:17:58.491563+0000","last_clean":"2024-03-04T16:17:58.491563+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:17:58.491563+0000","last_undegraded":"2024-03-04T16:17:58.491563+0000","last_fullsized":"2024-03-04T16:17:58.491563+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.370032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.370032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":269,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426085+0000","last_change":"2024-03-04T16:16:02.539110+0000","last_active":"2024-03-04T16:17:52.426085+0000","last_peered":"2024-03-04T16:17:52.426085+0000","last_clean":"2024-03-04T16:17:52.426085+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:17:52.426085+0000","last_undegraded":"2024-03-04T16:17:52.426085+0000","last_fullsized":"2024-03-04T16:17:52.426085+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.538977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.538977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":650,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426319+0000","last_change":"2024-03-04T16:17:01.305039+0000","last_active":"2024-03-04T16:17:52.426319+0000","last_peered":"2024-03-04T16:17:52.426319+0000","last_clean":"2024-03-04T16:17:52.426319+0000","last_became_active":"2024-03-04T16:10:29.325635+0000","last_became_peered":"2024-03-04T16:10:29.325635+0000","last_unstale":"2024-03-04T16:17:52.426319+0000","last_undegraded":"2024-03-04T16:17:52.426319+0000","last_fullsized":"2024-03-04T16:17:52.426319+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:17:01.304925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.304925+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:18:07.947920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"315'5905","reported_seq":5953,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.672025+0000","last_change":"2024-03-04T16:16:53.661912+0000","last_active":"2024-03-04T16:17:58.672025+0000","last_peered":"2024-03-04T16:17:58.672025+0000","last_clean":"2024-03-04T16:17:58.672025+0000","last_became_active":"2024-03-04T16:16:53.661564+0000","last_became_peered":"2024-03-04T16:16:53.661564+0000","last_unstale":"2024-03-04T16:17:58.672025+0000","last_undegraded":"2024-03-04T16:17:58.672025+0000","last_fullsized":"2024-03-04T16:17:58.672025+0000","mapping_epoch":296,"log_start":"305'2100","ondisk_log_start":"305'2100","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3805,"log_dups_size":0,"ondisk_log_size":3805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386990080,"num_objects":5905,"num_object_clones":0,"num_object_copies":17715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5905,"num_write_kb":377920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":543,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.431119+0000","last_change":"2024-03-04T16:15:47.547158+0000","last_active":"2024-03-04T16:17:52.431119+0000","last_peered":"2024-03-04T16:17:52.431119+0000","last_clean":"2024-03-04T16:17:52.431119+0000","last_became_active":"2024-03-04T16:10:17.474669+0000","last_became_peered":"2024-03-04T16:10:17.474669+0000","last_unstale":"2024-03-04T16:17:52.431119+0000","last_undegraded":"2024-03-04T16:17:52.431119+0000","last_fullsized":"2024-03-04T16:17:52.431119+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:47.547022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:15:47.547022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"192'10","reported_seq":469,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.429734+0000","last_change":"2024-03-04T16:17:44.249481+0000","last_active":"2024-03-04T16:17:52.429734+0000","last_peered":"2024-03-04T16:17:52.429734+0000","last_clean":"2024-03-04T16:17:52.429734+0000","last_became_active":"2024-03-04T16:10:17.468025+0000","last_became_peered":"2024-03-04T16:10:17.468025+0000","last_unstale":"2024-03-04T16:17:52.429734+0000","last_undegraded":"2024-03-04T16:17:52.429734+0000","last_fullsized":"2024-03-04T16:17:52.429734+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:17:44.249367+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.249367+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:09.533849+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":555,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-04T16:17:58.490639+0000","last_change":"2024-03-04T16:14:59.774268+0000","last_active":"2024-03-04T16:17:58.490639+0000","last_peered":"2024-03-04T16:17:58.490639+0000","last_clean":"2024-03-04T16:17:58.490639+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:17:58.490639+0000","last_undegraded":"2024-03-04T16:17:58.490639+0000","last_fullsized":"2024-03-04T16:17:58.490639+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:14:59.774151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"240'20","reported_seq":593,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-04T16:17:52.426816+0000","last_change":"2024-03-04T16:15:55.587146+0000","last_active":"2024-03-04T16:17:52.426816+0000","last_peered":"2024-03-04T16:17:52.426816+0000","last_clean":"2024-03-04T16:17:52.426816+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:17:52.426816+0000","last_undegraded":"2024-03-04T16:17:52.426816+0000","last_fullsized":"2024-03-04T16:17:52.426816+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:15:55.587052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:55.587052+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":6107103232,"num_objects":93187,"num_object_clones":0,"num_object_copies":279561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93187,"num_write_kb":5963968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18309840896,"data_stored":18309840896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65687,"ondisk_log_size":65687,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":907,"num_read_kb":902,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":233,"num_bytes_recovered":5050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":383,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070651,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2822128,"kb_used_data":2213416,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":90923024,"statfs":{"total":95995035648,"available":93105176576,"internally_reserved":0,"allocated":2266537984,"data_stored":2266028330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":623311306},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364991,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2186560,"kb_used_data":1493176,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":91558592,"statfs":{"total":95995035648,"available":93755998208,"internally_reserved":0,"allocated":1529012224,"data_stored":1528485275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":710015429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4282104,"kb_used_data":3697008,"kb_used_omap":1,"kb_used_meta":585086,"kb_avail":89463048,"statfs":{"total":95995035648,"available":91610161152,"internally_reserved":0,"allocated":3785736192,"data_stored":3785218330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":599128517},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411539,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149902,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3799028,"kb_used_data":2973740,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":89946124,"statfs":{"total":95995035648,"available":92104830976,"internally_reserved":0,"allocated":3045109760,"data_stored":3044595139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":845085128},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953673,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5355488,"kb_used_data":4778712,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":88389664,"statfs":{"total":95995035648,"available":90511015936,"internally_reserved":0,"allocated":4893401088,"data_stored":4892905691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626629,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3290392,"kb_used_data":2597008,"kb_used_omap":1,"kb_used_meta":693374,"kb_avail":90454760,"statfs":{"total":95995035648,"available":92625674240,"internally_reserved":0,"allocated":2659336192,"data_stored":2658848763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":710015430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3832676352,"data_stored":3832676352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1536098304,"data_stored":1536098304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3071868928,"data_stored":3071868928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4914085888,"data_stored":4914085888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2675769344,"data_stored":2675769344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2279342080,"data_stored":2279342080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:18:02.690 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-04T16:18:02.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-04T16:18:02.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:02.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: from='client.15825 v1:172.21.15.38:0/2254550611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[38113]: pgmap v679: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 638 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:18:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:18:02.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:18:02.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:18:02.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: from='client.15825 v1:172.21.15.38:0/2254550611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:18:02.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:02 smithi038 ceph-mon[29779]: pgmap v679: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 638 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-04T16:18:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.781 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:02.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: from='client.15825 v1:172.21.15.38:0/2254550611' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:18:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:02 smithi176 ceph-mon[33619]: pgmap v679: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 638 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-04T16:18:02.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:02.918 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:02.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:03.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 59 16 99471 99455 105.343 82.3125 0.0186043 0.00948829 2024-03-04T16:18:03.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:03.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:18:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.091 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:03.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:03.167 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:03.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:03.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:03.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:03.364 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:03.466 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:03.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:03.525 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:03.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:03.542 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:03.542 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:03.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:03.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:03.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:03.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:03.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:03.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:03.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:03.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:04.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:18:04.006875+0000 min lat: 0.00125439 max lat: 0.218493 avg lat: 0.00953496 2024-03-04T16:18:04.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:18:04.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 60 15 100643 100628 104.809 73.3125 0.0478602 0.00953496 2024-03-04T16:18:04.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:04.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:04.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.210 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:04.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:04.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.348 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:04.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:04.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:04.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:04.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:04.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:04 smithi176 ceph-mon[33619]: pgmap v680: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-04T16:18:04.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:04.731 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:04.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:04 smithi038 ceph-mon[29779]: pgmap v680: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-04T16:18:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:04 smithi038 ceph-mon[38113]: pgmap v680: 148 pgs: 148 active+clean; 6.0 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-04T16:18:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.779 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:04.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:04.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:04.915 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:04.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:04.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:05.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 61 16 102010 101994 104.49 85.375 0.0162525 0.00956581 2024-03-04T16:18:05.017 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.059 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:05.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:05.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:05.303 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:05.337 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:05.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:05.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:05.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:05.584 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:05.619 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:05.721 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:05.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.767 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:05 smithi176 ceph-mon[33619]: 4.13 deep-scrub starts 2024-03-04T16:18:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:05 smithi176 ceph-mon[33619]: 4.13 deep-scrub ok 2024-03-04T16:18:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:05 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:18:05.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:05 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:18:05.869 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:05.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:05.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:05.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[29779]: 4.13 deep-scrub starts 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[29779]: 4.13 deep-scrub ok 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[38113]: 4.13 deep-scrub starts 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[38113]: 4.13 deep-scrub ok 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:18:05.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:05 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:18:06.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 62 16 103350 103334 104.155 83.75 0.0069566 0.00959647 2024-03-04T16:18:06.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:06.052 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:06.155 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.204 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:06.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:06.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:06.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:06.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.461 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:06.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:06.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:06.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:06.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:06.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:06.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:06.855 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: osdmap e317: 8 total, 8 up, 6 in 2024-03-04T16:18:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: pgmap v682: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 715 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:18:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:18:06.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:06 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:18:06.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:06.884 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: osdmap e317: 8 total, 8 up, 6 in 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: pgmap v682: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 715 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: osdmap e317: 8 total, 8 up, 6 in 2024-03-04T16:18:06.885 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: pgmap v682: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 715 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-04T16:18:06.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:18:06.886 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:06 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:18:06.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:06.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:06.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:07.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 63 16 104399 104383 103.543 65.5625 0.0139493 0.00965331 2024-03-04T16:18:07.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:07.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:07.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:07.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:07.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:07.413 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:07.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:07.546 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:07.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:07.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:07.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:07.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:07.961 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:07.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:07.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:07 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:08.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 64 16 105666 105650 103.162 79.1875 0.00858526 0.00968926 2024-03-04T16:18:08.064 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:08.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:08.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:07 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:18:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:07 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:18:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:07 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:08.189 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:08.226 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:08.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:08.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:08.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:08.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:08.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:08.499 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:08.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:08.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:08.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:08.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:08.865 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:08.902 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:08.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[29779]: pgmap v683: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 814 op/s 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:18:08.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:18:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[38113]: pgmap v683: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 814 op/s 2024-03-04T16:18:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:09.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:09.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 65 16 106654 106638 102.525 61.75 0.0154062 0.00974917 2024-03-04T16:18:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.044 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:09.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:08 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:18:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:08 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:18:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:08 smithi176 ceph-mon[33619]: pgmap v683: 148 pgs: 148 active+clean; 6.1 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 814 op/s 2024-03-04T16:18:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:09.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:09.183 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:09.284 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:09.308 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:09.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:09.444 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:09.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:09.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:09.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:09.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:09.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:09.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:09.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:09 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:18:09.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:09 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:18:09.856 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:09 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:18:09.856 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:09 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:18:09.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:09.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:09.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:09.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:09.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:10.006 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 66 16 108091 108075 102.332 89.8125 0.00564455 0.00976689 2024-03-04T16:18:10.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:09 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:18:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:09 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:18:10.136 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:10.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:10.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:10.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:10.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.419 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:10.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:10.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:10.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:10.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:10.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:10.695 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:10.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:10.844 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:10.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:10.946 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:10.949 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 3.3 deep-scrub starts 2024-03-04T16:18:10.949 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 3.3 deep-scrub ok 2024-03-04T16:18:10.949 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 6.1 deep-scrub starts 2024-03-04T16:18:10.949 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: 6.1 deep-scrub ok 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: pgmap v684: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 3.3 deep-scrub starts 2024-03-04T16:18:10.950 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 3.3 deep-scrub ok 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 6.1 deep-scrub starts 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: 6.1 deep-scrub ok 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: pgmap v684: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:10.951 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:10 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:10.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:10.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:11.007 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 67 15 109548 109533 102.164 91.125 0.0156487 0.00978369 2024-03-04T16:18:11.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 3.3 deep-scrub starts 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 3.3 deep-scrub ok 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 6.1 deep-scrub starts 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:18:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: 6.1 deep-scrub ok 2024-03-04T16:18:11.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: pgmap v684: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-04T16:18:11.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:11.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:10 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:11.122 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:11.206 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:11.224 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:11.288 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:11.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:11.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:11.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:11.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.564 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:11.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:11.665 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.704 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:11.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:11.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:11.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[29779]: osdmap e318: 8 total, 8 up, 6 in 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[38113]: osdmap e318: 8 total, 8 up, 6 in 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:18:11.855 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:11 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:18:11.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:11.964 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:18:11.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:11.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:12.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:18:12.007 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 68 16 110912 110896 101.914 85.1875 0.00721848 0.00980754 2024-03-04T16:18:12.047 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:18:12.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:12.091 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:18:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:11 smithi176 ceph-mon[33619]: osdmap e318: 8 total, 8 up, 6 in 2024-03-04T16:18:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:11 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:18:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:11 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:18:12.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:12.136 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:18:12.178 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:18:12.218 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:18:12.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:12.261 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:18:12.264 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:12.270 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:18:12.270 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:18:12.270 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:18:12.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:18:12.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:18:12.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:18:12.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:18:12.271 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:18:12.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:12.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:12.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:12.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:12.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.678 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:12.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:12.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.817 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:12.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:12.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:12.965 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:12.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:12.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: pgmap v686: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2480619677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:18:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: pgmap v686: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-04T16:18:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:18:12.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2480619677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:13.007 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 69 16 112334 112318 101.725 88.875 0.0070407 0.00982593 2024-03-04T16:18:13.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:13.103 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: pgmap v686: 148 pgs: 148 active+clean; 6.5 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:18:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2480619677' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:13.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:13.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.379 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:13.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:13.443 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:13.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:13.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:13.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:13.618 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:13.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[29779]: 6.18 deep-scrub starts 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[29779]: 6.18 deep-scrub ok 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:18:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:18:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[38113]: 6.18 deep-scrub starts 2024-03-04T16:18:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[38113]: 6.18 deep-scrub ok 2024-03-04T16:18:13.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:13.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:13.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:13.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:13.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:13.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:13.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:14.007 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 70 16 113815 113799 101.594 92.5625 0.00689272 0.0098388 2024-03-04T16:18:14.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:14.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:18:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:18:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-mon[33619]: 6.18 deep-scrub starts 2024-03-04T16:18:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-mon[33619]: 6.18 deep-scrub ok 2024-03-04T16:18:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:14.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:14.244 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:14.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.386 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:14.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:14.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:14.517 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:14.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:14.658 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:14.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:14.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:14.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:14.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:14.920 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:14.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.14 deep-scrub starts 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.14 deep-scrub ok 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.18 deep-scrub starts 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: pgmap v687: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 5.18 deep-scrub ok 2024-03-04T16:18:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[29779]: 4.c deep-scrub starts 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.14 deep-scrub starts 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.14 deep-scrub ok 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.18 deep-scrub starts 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: pgmap v687: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 5.18 deep-scrub ok 2024-03-04T16:18:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:14 smithi038 ceph-mon[38113]: 4.c deep-scrub starts 2024-03-04T16:18:15.007 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 71 15 115139 115124 101.33 82.8125 0.0039265 0.00986298 2024-03-04T16:18:15.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.059 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:15.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.14 deep-scrub starts 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.14 deep-scrub ok 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:18:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:18:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.18 deep-scrub starts 2024-03-04T16:18:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: pgmap v687: 148 pgs: 148 active+clean; 6.7 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-04T16:18:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 5.18 deep-scrub ok 2024-03-04T16:18:15.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:14 smithi176 ceph-mon[33619]: 4.c deep-scrub starts 2024-03-04T16:18:15.149 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:15.160 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:15.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:15.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:15.358 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:15.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:15.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:15.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:15.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:15.626 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:15.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:15.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:15.772 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:15.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:15.916 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:15.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:15.917 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[29779]: 4.c deep-scrub ok 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:18:15.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:15 smithi038 ceph-mon[38113]: 4.c deep-scrub ok 2024-03-04T16:18:15.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:18:16.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 72 16 116485 116469 101.089 84.0625 0.00648434 0.00988763 2024-03-04T16:18:16.018 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:16.052 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:15 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:18:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:15 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:18:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:15 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:18:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:15 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:18:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:15 smithi176 ceph-mon[33619]: 4.c deep-scrub ok 2024-03-04T16:18:16.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:16.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:16.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.349 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:16.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:16.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:16.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:16.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.625 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:16.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:16.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:16.781 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:16.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:16.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:16.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:16.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:18:16.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:18:16.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: pgmap v688: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: pgmap v688: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-04T16:18:16.955 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:18:16.956 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:18:16.956 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:18:16.956 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:18:16.956 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-04T16:18:17.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 73 15 117834 117819 100.86 84.375 0.0248618 0.00991004 2024-03-04T16:18:17.031 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.066 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:17.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: pgmap v688: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:18:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-04T16:18:17.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:17.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:17.346 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 10 pg_num to 26 2024-03-04T16:18:17.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:17.473 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:17.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:17.677 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:17.693 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:18:17.693 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:18:17.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:17.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.820 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:17.841 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: osdmap e319: 8 total, 8 up, 6 in 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:18:17.842 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-04T16:18:17.843 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: osdmap e319: 8 total, 8 up, 6 in 2024-03-04T16:18:17.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:18:17.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:17.844 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:17 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:17.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:17.965 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:17.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:18.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 74 16 119121 119105 100.583 80.375 0.0108234 0.00993768 2024-03-04T16:18:18.066 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:18.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:18:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1262172071' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: osdmap e319: 8 total, 8 up, 6 in 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:18.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:17 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:18.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:18.243 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:18.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:18.346 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:18:18.400 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:18.502 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:18.545 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:18.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:18:18.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:18.670 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:18.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:18.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:18.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:18.921 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:18.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: pgmap v690: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-04T16:18:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[29779]: osdmap e320: 8 total, 8 up, 6 in 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: pgmap v690: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-04T16:18:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:18 smithi038 ceph-mon[38113]: osdmap e320: 8 total, 8 up, 6 in 2024-03-04T16:18:19.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 75 16 119632 119616 99.6679 31.9375 0.00577904 0.00994948 2024-03-04T16:18:19.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:19.046 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: pgmap v690: 148 pgs: 148 active+clean; 6.9 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 855 op/s 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3487395846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-04T16:18:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:18 smithi176 ceph-mon[33619]: osdmap e320: 8 total, 8 up, 6 in 2024-03-04T16:18:19.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:19.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:19.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:19.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:19.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:19.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:19.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:19.577 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:19.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.719 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:19.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:19.794 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:19.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:19.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:19.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:19.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:20.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 76 16 120754 120738 99.279 70.125 0.0162218 0.0100685 2024-03-04T16:18:20.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:20.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:20.154 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:20.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:20.279 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:20.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:20.405 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:20.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:20.530 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:20.631 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:20.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:20.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[29779]: osdmap e321: 8 total, 8 up, 6 in 2024-03-04T16:18:20.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.722 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.722 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[38113]: osdmap e321: 8 total, 8 up, 6 in 2024-03-04T16:18:20.723 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.723 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:20 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.758 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:20.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:20 smithi176 ceph-mon[33619]: osdmap e321: 8 total, 8 up, 6 in 2024-03-04T16:18:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:20 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:20 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-04T16:18:20.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:20.912 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:20.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:21.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 77 16 121335 121319 98.4612 36.3125 0.00180526 0.0100685 2024-03-04T16:18:21.014 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:21.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:21.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:21.085 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:21.140 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:21.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:21.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:21.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:21.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:18:21.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:21.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:21.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:21.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[29779]: pgmap v693: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-04T16:18:21.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-04T16:18:21.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[29779]: osdmap e322: 8 total, 8 up, 6 in 2024-03-04T16:18:21.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[38113]: pgmap v693: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-04T16:18:21.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-04T16:18:21.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:21 smithi038 ceph-mon[38113]: osdmap e322: 8 total, 8 up, 6 in 2024-03-04T16:18:21.592 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:21.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:21.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:21.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:21.804 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:18:21.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:21.824 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:18:21.844 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:18:21.846 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:21 smithi176 ceph-mon[33619]: pgmap v693: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-04T16:18:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:21 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-04T16:18:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:21 smithi176 ceph-mon[33619]: osdmap e322: 8 total, 8 up, 6 in 2024-03-04T16:18:21.864 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:18:21.884 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:18:21.905 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:18:21.926 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:18:21.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:21.951 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:18:21.953 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:18:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:21.975 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:21.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:22.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 78 16 121335 121319 97.1989 0 - 0.0100685 2024-03-04T16:18:22.076 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.104 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:22.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:22.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:22.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:22.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.355 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:22.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:22.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:22.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[38113]: osdmap e323: 8 total, 8 up, 6 in 2024-03-04T16:18:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[38113]: pgmap v696: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-04T16:18:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/349261408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:22.495 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[38113]: osdmap e324: 8 total, 8 up, 6 in 2024-03-04T16:18:22.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[29779]: osdmap e323: 8 total, 8 up, 6 in 2024-03-04T16:18:22.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[29779]: pgmap v696: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-04T16:18:22.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/349261408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:22.496 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:22 smithi038 ceph-mon[29779]: osdmap e324: 8 total, 8 up, 6 in 2024-03-04T16:18:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.518 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:22.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:22.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:22.650 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:22.751 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.783 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:22.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:22 smithi176 ceph-mon[33619]: osdmap e323: 8 total, 8 up, 6 in 2024-03-04T16:18:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:22 smithi176 ceph-mon[33619]: pgmap v696: 158 pgs: 5 peering, 10 unknown, 143 active+clean; 7.3 GiB data, 26 GiB used, 511 GiB / 536 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-04T16:18:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/349261408' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:22 smithi176 ceph-mon[33619]: osdmap e324: 8 total, 8 up, 6 in 2024-03-04T16:18:22.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:22.919 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:22.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:23.008 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 79 16 122093 122077 96.5682 23.6875 0.00981723 0.0103512 2024-03-04T16:18:23.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:23.059 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:23.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.197 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:23.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:23.298 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:23.332 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:23.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:23.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:23.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:23.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:23.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 321'7487 2024-03-04T16:18:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7479 2024-03-04T16:18:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7446 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7552 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:23.610 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:23.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:23.722 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:23.742 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 321'7487 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7479 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7446 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7552 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:23.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 321'7487 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7479 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7446 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7552 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7600 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7526 2024-03-04T16:18:23.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7617 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 321'7696 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 321'7589 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:23.746 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:23.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:23.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:23.905 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:23.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:24.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:24.009 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:18:24.009673+0000 min lat: 0.00125439 max lat: 2.14381 avg lat: 0.0103596 2024-03-04T16:18:24.009 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:18:24.009 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 80 15 123536 123521 96.4891 90.25 0.00772286 0.0103596 2024-03-04T16:18:24.064 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:24.166 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:24.314 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:24.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:24.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:24.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:24.609 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:24.641 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:24 smithi038 ceph-mon[29779]: pgmap v698: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 3 peering, 4 active+remapped+backfilling, 2 unknown, 145 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 45535/353799 objects misplaced (12.870%); 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:18:24.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:24 smithi038 ceph-mon[38113]: pgmap v698: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 3 peering, 4 active+remapped+backfilling, 2 unknown, 145 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 45535/353799 objects misplaced (12.870%); 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:18:24.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:24.769 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:18:24.769 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:18:24.769 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-04T16:18:24.769 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-04T16:18:24.769 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-04T16:18:24.769 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-04T16:18:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:24 smithi176 ceph-mon[33619]: pgmap v698: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 3 peering, 4 active+remapped+backfilling, 2 unknown, 145 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 45535/353799 objects misplaced (12.870%); 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:18:24.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:24.961 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:24.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:25.009 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 81 16 124815 124799 96.2839 79.875 0.01363 0.0103813 2024-03-04T16:18:25.063 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:25.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:25.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:25.232 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:25.334 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:25.364 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:25.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:25.378 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:25.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.520 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:25.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:25.621 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:25.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[38113]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:25.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[29779]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:25.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:25.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:25.819 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:25 smithi176 ceph-mon[33619]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:25.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:25.956 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:25.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:26.009 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 82 16 126024 126008 96.0304 75.5625 0.0135043 0.0104086 2024-03-04T16:18:26.058 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:26.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:18:26.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:26.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:26.296 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:26.398 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:26.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:26.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:26 smithi038 ceph-mon[29779]: pgmap v699: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 34 MiB/s wr, 541 op/s; 60779/368121 objects misplaced (16.511%); 5.6 MiB/s, 89 objects/s recovering 2024-03-04T16:18:26.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:26 smithi038 ceph-mon[38113]: pgmap v699: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 34 MiB/s wr, 541 op/s; 60779/368121 objects misplaced (16.511%); 5.6 MiB/s, 89 objects/s recovering 2024-03-04T16:18:26.530 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:18:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:18:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:18:26.530 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:18:26.530 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:18:26.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:26.562 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:26.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:26.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:26.699 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:26.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:26.835 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:26 smithi176 ceph-mon[33619]: pgmap v699: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 34 MiB/s wr, 541 op/s; 60779/368121 objects misplaced (16.511%); 5.6 MiB/s, 89 objects/s recovering 2024-03-04T16:18:26.872 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:26.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:26.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:27.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 83 16 127117 127101 95.6962 68.3125 0.0186387 0.0104449 2024-03-04T16:18:27.029 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:27.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:27.164 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:27.265 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.308 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:27.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:27.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:27.443 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:27.545 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:27.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:27.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:27.679 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:27.712 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:27.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:18:27.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:18:27.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:27.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:18:27.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:18:27.720 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:27 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:27.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:27 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:18:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:27 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:18:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:27 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:27.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:27.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:27.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:28.004 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:28.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 84 16 128424 128408 95.5294 81.6875 0.0140906 0.0104641 2024-03-04T16:18:28.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:28.141 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:28.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:28.278 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:28.294 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:28.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:18:28.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.449 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:28.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:28.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:28.597 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:28 smithi038 ceph-mon[29779]: pgmap v700: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 703 B/s rd, 28 MiB/s wr, 446 op/s; 60779/368121 objects misplaced (16.511%); 4.6 MiB/s, 74 objects/s recovering 2024-03-04T16:18:28.597 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:28 smithi038 ceph-mon[38113]: pgmap v700: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 703 B/s rd, 28 MiB/s wr, 446 op/s; 60779/368121 objects misplaced (16.511%); 4.6 MiB/s, 74 objects/s recovering 2024-03-04T16:18:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:28.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:28.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:28.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:28 smithi176 ceph-mon[33619]: pgmap v700: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 703 B/s rd, 28 MiB/s wr, 446 op/s; 60779/368121 objects misplaced (16.511%); 4.6 MiB/s, 74 objects/s recovering 2024-03-04T16:18:28.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:28.912 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:28.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:28.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:29.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 85 16 129738 129722 95.3716 82.125 0.012517 0.0104807 2024-03-04T16:18:29.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:29.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:29.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:29.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:29.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:29.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:29.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:29.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:29.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:29.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:29.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:29.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:29.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:29.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:29.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:29.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:30.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:30.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 86 16 131017 131001 95.1921 79.9375 0.00770741 0.010501 2024-03-04T16:18:30.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:30.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:30.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:30.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:30.334 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:30.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:30.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:30.481 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:30.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:30.618 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:30.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:30.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:30.721 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:18:30.768 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:30.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:30.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:30.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:30.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:30.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:30 smithi038 ceph-mon[29779]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 59329/386913 objects misplaced (15.334%); 25 MiB/s, 404 objects/s recovering 2024-03-04T16:18:30.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:18:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:30 smithi038 ceph-mon[38113]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 59329/386913 objects misplaced (15.334%); 25 MiB/s, 404 objects/s recovering 2024-03-04T16:18:30.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:18:31.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:31.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 87 16 132243 132227 94.9786 76.625 0.0138944 0.0105244 2024-03-04T16:18:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.051 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:31.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:30 smithi176 ceph-mon[33619]: pgmap v701: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 59329/386913 objects misplaced (15.334%); 25 MiB/s, 404 objects/s recovering 2024-03-04T16:18:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:18:31.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:31.194 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:31.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:31.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:31.435 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:31.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:31.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:31.606 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:31.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:31.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:31.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:31 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:31 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:18:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:31 smithi176 ceph-mon[33619]: osdmap e325: 8 total, 8 up, 6 in 2024-03-04T16:18:31.893 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:31.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[29779]: osdmap e325: 8 total, 8 up, 6 in 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1795449495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:18:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:31 smithi038 ceph-mon[38113]: osdmap e325: 8 total, 8 up, 6 in 2024-03-04T16:18:31.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:32.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 88 16 133651 133635 94.8992 88 0.0094112 0.0105331 2024-03-04T16:18:32.040 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:32.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:32.177 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:32.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.329 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:32.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:32.430 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:32.467 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:32.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:32.614 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:32.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:32.758 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:32.860 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:32.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:32.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:32.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:32 smithi038 ceph-mon[29779]: pgmap v703: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 59329/386913 objects misplaced (15.334%); 23 MiB/s, 360 objects/s recovering 2024-03-04T16:18:32.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:32 smithi038 ceph-mon[38113]: pgmap v703: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 59329/386913 objects misplaced (15.334%); 23 MiB/s, 360 objects/s recovering 2024-03-04T16:18:32.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:33.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 89 16 134856 134840 94.6791 75.3125 0.00925592 0.0105576 2024-03-04T16:18:33.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:33.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:32 smithi176 ceph-mon[33619]: pgmap v703: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.9 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 59329/386913 objects misplaced (15.334%); 23 MiB/s, 360 objects/s recovering 2024-03-04T16:18:33.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.178 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:33.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:33.279 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:33.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:33.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:33.461 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:33.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:33.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:33.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:33.707 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:33.727 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:33.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:33.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:33.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:33.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:33.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:33 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:33.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:33 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:34.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total time run: 90.0089 2024-03-04T16:18:34.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total writes made: 135954 2024-03-04T16:18:34.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Write size: 65536 2024-03-04T16:18:34.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object size: 65536 2024-03-04T16:18:34.010 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Bandwidth (MB/sec): 94.4032 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Bandwidth: 33.4702 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max bandwidth (MB/sec): 222.562 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min bandwidth (MB/sec): 0 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average IOPS: 1510 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev IOPS: 535.523 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max IOPS: 3561 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min IOPS: 0 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average Latency(s): 0.0105889 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Latency(s): 0.0265988 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max latency(s): 2.14381 2024-03-04T16:18:34.011 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min latency(s): 0.00125439 2024-03-04T16:18:34.014 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Cleaning up (deleting benchmark objects) 2024-03-04T16:18:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:33 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:34.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:18:34.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:34.259 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:34.293 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:34.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:34.436 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:34.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:34.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:34.584 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:34.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:34.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:34.858 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:34.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:34.984 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:34.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:34.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:34.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:34 smithi038 ceph-mon[29779]: pgmap v704: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s; 57972/400977 objects misplaced (14.458%); 33 MiB/s, 521 objects/s recovering 2024-03-04T16:18:34.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:34 smithi038 ceph-mon[38113]: pgmap v704: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s; 57972/400977 objects misplaced (14.458%); 33 MiB/s, 521 objects/s recovering 2024-03-04T16:18:35.085 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:34 smithi176 ceph-mon[33619]: pgmap v704: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s; 57972/400977 objects misplaced (14.458%); 33 MiB/s, 521 objects/s recovering 2024-03-04T16:18:35.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:35.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:35.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:35.273 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:35.375 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.411 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:35.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:35.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:35.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:35.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:35.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:35.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.826 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:35.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:35.927 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:35.963 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:35.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[29779]: osdmap e326: 8 total, 8 up, 6 in 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:35 smithi038 ceph-mon[38113]: osdmap e326: 8 total, 8 up, 6 in 2024-03-04T16:18:36.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:36.104 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:35 smithi176 ceph-mon[33619]: osdmap e326: 8 total, 8 up, 6 in 2024-03-04T16:18:36.206 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:36.230 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:36.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:36.367 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:36.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:36.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:36.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:36.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:36.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:36.749 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:36.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:36.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:36.926 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:36.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:37.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:37.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.069 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:37.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:37.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:36 smithi176 ceph-mon[33619]: pgmap v706: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 57514/406218 objects misplaced (14.158%); 44 MiB/s, 703 objects/s recovering 2024-03-04T16:18:37.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:37.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:37.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:36 smithi038 ceph-mon[29779]: pgmap v706: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 57514/406218 objects misplaced (14.158%); 44 MiB/s, 703 objects/s recovering 2024-03-04T16:18:37.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:36 smithi038 ceph-mon[38113]: pgmap v706: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.2 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 57514/406218 objects misplaced (14.158%); 44 MiB/s, 703 objects/s recovering 2024-03-04T16:18:37.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:37.350 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:37.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:37.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:37.593 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:37.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:37.735 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:37.761 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:37.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:37.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:37.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:38.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:38.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:38.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.176 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:38.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:38.277 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:38.321 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:38.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:38.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:38.607 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:38.646 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:38.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.772 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:38.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:38.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:38.904 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:38.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:39.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:39.048 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:39.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:39.174 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:39.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[29779]: pgmap v707: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s; 57514/406218 objects misplaced (14.158%); 22 MiB/s, 353 objects/s recovering 2024-03-04T16:18:39.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[38113]: pgmap v707: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s; 57514/406218 objects misplaced (14.158%); 22 MiB/s, 353 objects/s recovering 2024-03-04T16:18:39.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:39.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:39.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:39.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:39 smithi176 ceph-mon[33619]: pgmap v707: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.3 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 639 B/s rd, 50 MiB/s wr, 804 op/s; 57514/406218 objects misplaced (14.158%); 22 MiB/s, 353 objects/s recovering 2024-03-04T16:18:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:39 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:39 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:39 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:39 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:39.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:39.453 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:39.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:39 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:39.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:39.578 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:39.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:39.724 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:39.765 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:18:39.765 INFO:teuthology.orchestra.run.smithi038.stdout: "success": "" 2024-03-04T16:18:39.765 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:18:39.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:39.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:39.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:39.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:39.986 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:39.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:40.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:40.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:40.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:40.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:40.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:40.355 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:40.391 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:40.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:40 smithi038 ceph-mon[29779]: osdmap e327: 8 total, 8 up, 6 in 2024-03-04T16:18:40.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:40 smithi038 ceph-mon[38113]: osdmap e327: 8 total, 8 up, 6 in 2024-03-04T16:18:40.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:40.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:40.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:40 smithi176 ceph-mon[33619]: osdmap e327: 8 total, 8 up, 6 in 2024-03-04T16:18:40.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:40.671 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:40.773 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:40.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:40.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:40.940 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:40.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:41.042 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.075 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:41.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:41.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.200 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:41.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:41.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:41.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:41.447 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:41.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:41.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:41 smithi038 ceph-mon[29779]: pgmap v709: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.54k op/s; 51335/393039 objects misplaced (13.061%); 43 MiB/s, 682 objects/s recovering 2024-03-04T16:18:41.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:41 smithi038 ceph-mon[38113]: pgmap v709: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.54k op/s; 51335/393039 objects misplaced (13.061%); 43 MiB/s, 682 objects/s recovering 2024-03-04T16:18:41.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:41.607 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:41 smithi176 ceph-mon[33619]: pgmap v709: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 1.4 KiB/s rd, 56 MiB/s wr, 1.54k op/s; 51335/393039 objects misplaced (13.061%); 43 MiB/s, 682 objects/s recovering 2024-03-04T16:18:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.611 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:41.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:41.711 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:41.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:41.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:41.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:41.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:42.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:42.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:42.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:42.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:42.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:42.394 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:42.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:42.501 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:42.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:42 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:42 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:42.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:42 smithi176 ceph-mon[33619]: osdmap e328: 8 total, 8 up, 6 in 2024-03-04T16:18:42.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:42.661 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:42.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[29779]: osdmap e328: 8 total, 8 up, 6 in 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:42 smithi038 ceph-mon[38113]: osdmap e328: 8 total, 8 up, 6 in 2024-03-04T16:18:42.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:42.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:42.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:42.935 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:42.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:43.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:43.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:43.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:43.209 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:43.311 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:43.334 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:43.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:43.473 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:43.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:43 smithi176 ceph-mon[33619]: pgmap v711: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 893 B/s rd, 6.8 MiB/s wr, 856 op/s; 51335/393039 objects misplaced (13.061%); 24 MiB/s, 383 objects/s recovering 2024-03-04T16:18:43.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:43.610 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:43.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:43.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:43 smithi038 ceph-mon[29779]: pgmap v711: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 893 B/s rd, 6.8 MiB/s wr, 856 op/s; 51335/393039 objects misplaced (13.061%); 24 MiB/s, 383 objects/s recovering 2024-03-04T16:18:43.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:43 smithi038 ceph-mon[38113]: pgmap v711: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 8.0 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 893 B/s rd, 6.8 MiB/s wr, 856 op/s; 51335/393039 objects misplaced (13.061%); 24 MiB/s, 383 objects/s recovering 2024-03-04T16:18:43.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:43.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:43.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:43.892 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:43.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:43.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:44.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:44.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:44.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:44.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:44.269 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:44.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:44.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:44.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:44.432 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:44.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:44.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:44.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:44.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:44.573 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:44.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:44.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.705 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:44.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:44.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:44.837 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:44.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:44.973 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:44.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:45.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:45.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:45.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:45.238 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:45.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:45.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.369 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:45.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:45.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:45.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:45.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:45.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:45 smithi176 ceph-mon[33619]: pgmap v712: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.8 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.19k op/s; 47270/382119 objects misplaced (12.370%); 34 MiB/s, 549 objects/s recovering 2024-03-04T16:18:45.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:45.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:45.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:45 smithi038 ceph-mon[29779]: pgmap v712: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.8 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.19k op/s; 47270/382119 objects misplaced (12.370%); 34 MiB/s, 549 objects/s recovering 2024-03-04T16:18:45.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:45 smithi038 ceph-mon[38113]: pgmap v712: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.8 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.19k op/s; 47270/382119 objects misplaced (12.370%); 34 MiB/s, 549 objects/s recovering 2024-03-04T16:18:45.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:45.870 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:45.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:45.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:45.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:46.035 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:46.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:46.172 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:46.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:46.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:46.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.441 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:46.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.585 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:46.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:46.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:46.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.858 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:46.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:46.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:46.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:47.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.131 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:47.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:47.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:47.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:47.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:47.402 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:47.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:47.542 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:47.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:47 smithi176 ceph-mon[33619]: pgmap v713: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.37k op/s; 45932/377739 objects misplaced (12.160%); 38 MiB/s, 601 objects/s recovering 2024-03-04T16:18:47.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:47.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:47.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:47.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:47 smithi038 ceph-mon[29779]: pgmap v713: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.37k op/s; 45932/377739 objects misplaced (12.160%); 38 MiB/s, 601 objects/s recovering 2024-03-04T16:18:47.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:47 smithi038 ceph-mon[38113]: pgmap v713: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 5.8 MiB/s wr, 1.37k op/s; 45932/377739 objects misplaced (12.160%); 38 MiB/s, 601 objects/s recovering 2024-03-04T16:18:47.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:47.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:47.917 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:47.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:47.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:48.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:48.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:48.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:48.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:48.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:48.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:48.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:48.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:48.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:48.621 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:48.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:48.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:48.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:48.895 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:48.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:48.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:49.027 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:49.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:49.171 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:49.273 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:49.307 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:49.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:49.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:49.432 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:49.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.578 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:49 smithi176 ceph-mon[33619]: pgmap v714: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 606 B/s rd, 0 B/s wr, 605 op/s; 45932/377739 objects misplaced (12.160%); 16 MiB/s, 258 objects/s recovering 2024-03-04T16:18:49.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:49.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:49 smithi038 ceph-mon[29779]: pgmap v714: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 606 B/s rd, 0 B/s wr, 605 op/s; 45932/377739 objects misplaced (12.160%); 16 MiB/s, 258 objects/s recovering 2024-03-04T16:18:49.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:49 smithi038 ceph-mon[38113]: pgmap v714: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.7 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 606 B/s rd, 0 B/s wr, 605 op/s; 45932/377739 objects misplaced (12.160%); 16 MiB/s, 258 objects/s recovering 2024-03-04T16:18:49.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:49.865 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:49.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:49.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:50.005 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:50.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:50.142 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:50.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.268 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:50.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:50.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:50.402 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:50.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:50.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:50.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:50.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.680 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:50.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:50.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:50.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:50.914 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:50.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:50.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:51.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:51.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:51.191 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:51.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:51.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:51.366 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:51.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:51 smithi038 ceph-mon[29779]: pgmap v715: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 39977/361446 objects misplaced (11.060%); 29 MiB/s, 467 objects/s recovering 2024-03-04T16:18:51.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:51 smithi038 ceph-mon[29779]: osdmap e329: 8 total, 8 up, 6 in 2024-03-04T16:18:51.456 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:51 smithi038 ceph-mon[38113]: pgmap v715: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 39977/361446 objects misplaced (11.060%); 29 MiB/s, 467 objects/s recovering 2024-03-04T16:18:51.457 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:51 smithi038 ceph-mon[38113]: osdmap e329: 8 total, 8 up, 6 in 2024-03-04T16:18:51.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:51.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:51.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:51 smithi176 ceph-mon[33619]: pgmap v715: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s; 39977/361446 objects misplaced (11.060%); 29 MiB/s, 467 objects/s recovering 2024-03-04T16:18:51.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:51 smithi176 ceph-mon[33619]: osdmap e329: 8 total, 8 up, 6 in 2024-03-04T16:18:51.649 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:51.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:51.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:51.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:51.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:51.935 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:51.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:52.036 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:52.069 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:52.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:52.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:52.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:52.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.342 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:52.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:52.443 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:52.489 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:18:52.489 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:52 smithi038 ceph-mon[29779]: osdmap e330: 8 total, 8 up, 6 in 2024-03-04T16:18:52.489 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:52 smithi038 ceph-mon[38113]: osdmap e330: 8 total, 8 up, 6 in 2024-03-04T16:18:52.532 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:18:52.575 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:18:52.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:52 smithi176 ceph-mon[33619]: osdmap e330: 8 total, 8 up, 6 in 2024-03-04T16:18:52.623 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:52.624 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:18:52.665 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:18:52.705 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:18:52.725 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:52.747 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:18:52.756 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:52.789 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:18:52.793 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:18:52.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:52.894 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:52.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:52.996 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:53.023 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:53.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:53.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:53.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:53.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:53.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:53.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:53.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:53.559 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:53.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-mon[33619]: pgmap v718: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 39977/361446 objects misplaced (11.060%); 23 MiB/s, 363 objects/s recovering 2024-03-04T16:18:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-mon[33619]: osdmap e331: 8 total, 8 up, 6 in 2024-03-04T16:18:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3672134647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:53.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:53.609 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:18:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:18:53.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:53.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:53.736 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:18:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:18:53.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[38113]: pgmap v718: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 39977/361446 objects misplaced (11.060%); 23 MiB/s, 363 objects/s recovering 2024-03-04T16:18:53.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[38113]: osdmap e331: 8 total, 8 up, 6 in 2024-03-04T16:18:53.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3672134647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:53.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:53.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:53.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[29779]: pgmap v718: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 7.3 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 861 op/s; 39977/361446 objects misplaced (11.060%); 23 MiB/s, 363 objects/s recovering 2024-03-04T16:18:53.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[29779]: osdmap e331: 8 total, 8 up, 6 in 2024-03-04T16:18:53.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3672134647' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:18:53.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:18:53.738 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:18:53.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:53.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:53.952 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:53.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:53.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:54.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:54.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:54.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:54.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:54.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:54.564 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:54.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:54.700 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:54 smithi038 ceph-mon[29779]: osdmap e332: 8 total, 8 up, 6 in 2024-03-04T16:18:54.700 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:54 smithi038 ceph-mon[29779]: pgmap v721: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 149 active+clean; 7.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 629 op/s; 38376/353904 objects misplaced (10.844%); 15 MiB/s, 235 objects/s recovering 2024-03-04T16:18:54.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:54 smithi038 ceph-mon[38113]: osdmap e332: 8 total, 8 up, 6 in 2024-03-04T16:18:54.700 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:54 smithi038 ceph-mon[38113]: pgmap v721: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 149 active+clean; 7.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 629 op/s; 38376/353904 objects misplaced (10.844%); 15 MiB/s, 235 objects/s recovering 2024-03-04T16:18:54.711 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:54.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.849 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:54.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:54 smithi176 ceph-mon[33619]: osdmap e332: 8 total, 8 up, 6 in 2024-03-04T16:18:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:54 smithi176 ceph-mon[33619]: pgmap v721: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfill_wait, 4 active+remapped+backfilling, 149 active+clean; 7.2 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 629 op/s; 38376/353904 objects misplaced (10.844%); 15 MiB/s, 235 objects/s recovering 2024-03-04T16:18:54.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:54.996 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:54.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:54.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:55.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:55.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:55.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:55.239 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:55.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:55.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:55.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:55.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[29779]: osdmap e333: 8 total, 8 up, 6 in 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:55 smithi038 ceph-mon[38113]: osdmap e333: 8 total, 8 up, 6 in 2024-03-04T16:18:55.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:55.538 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:55.619 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:18:55.619 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:18:55.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:55.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:55.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:55.772 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:55.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:18:55.816 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:55 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:18:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:55 smithi176 ceph-mon[33619]: osdmap e333: 8 total, 8 up, 6 in 2024-03-04T16:18:55.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:55.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:55.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:55.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:56.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:56.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:56.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:56.236 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7] out_osds: [2, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:18:56.236 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:18:56.237 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-03-04T16:18:56.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd in 2 2024-03-04T16:18:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:56.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:56.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:56.402 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:18:56.504 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:56.536 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:56.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:56.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[29779]: pgmap v723: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 32717/346812 objects misplaced (9.434%); 47 MiB/s, 748 objects/s recovering 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:18:56.727 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[38113]: pgmap v723: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 32717/346812 objects misplaced (9.434%); 47 MiB/s, 748 objects/s recovering 2024-03-04T16:18:56.728 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:56 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:18:56.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.818 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:56.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:56 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:18:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:56 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:18:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:56 smithi176 ceph-mon[33619]: pgmap v723: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 32717/346812 objects misplaced (9.434%); 47 MiB/s, 748 objects/s recovering 2024-03-04T16:18:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:56 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:18:56.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:56.957 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:56.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:57.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.103 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:57.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:18:57.242 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:18:57.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:57.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:57.538 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:57.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:57.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:57.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:18:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[29779]: osdmap e334: 8 total, 8 up, 6 in 2024-03-04T16:18:57.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:57.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:18:57.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:57 smithi038 ceph-mon[38113]: osdmap e334: 8 total, 8 up, 6 in 2024-03-04T16:18:57.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.821 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:57 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:18:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:57 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3686792875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:18:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:57 smithi176 ceph-mon[33619]: osdmap e334: 8 total, 8 up, 6 in 2024-03-04T16:18:57.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:18:57.951 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:57.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:57.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:18:58.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:58.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:58.190 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:58.324 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:58.362 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:58.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:58.495 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:58.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:18:58.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:18:58.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:58.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:18:58 smithi038 ceph-mon[29779]: pgmap v725: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 929 op/s; 32717/346812 objects misplaced (9.434%); 38 MiB/s, 611 objects/s recovering 2024-03-04T16:18:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:18:58 smithi038 ceph-mon[38113]: pgmap v725: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 929 op/s; 32717/346812 objects misplaced (9.434%); 38 MiB/s, 611 objects/s recovering 2024-03-04T16:18:58.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:18:58 smithi176 ceph-mon[33619]: pgmap v725: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 peering, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.0 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 974 B/s rd, 0 B/s wr, 929 op/s; 32717/346812 objects misplaced (9.434%); 38 MiB/s, 611 objects/s recovering 2024-03-04T16:18:58.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:58.904 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:58.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:58.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:18:59.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:18:59.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:59.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:18:59.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:18:59.198 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:18:59.300 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.338 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:59.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:59.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:18:59.474 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:18:59.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:18:59.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:18:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:18:59.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:59.747 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:59.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:59.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:18:59.887 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:18:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:18:59.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:18:59.969 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:59.977 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:18:59.989 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:00.047 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:00.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:00.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:00.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:00.313 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:00.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:00.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:00.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:00.592 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:00.694 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:00.727 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:00.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:00.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[29779]: osdmap e335: 8 total, 8 up, 6 in 2024-03-04T16:19:00.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[29779]: pgmap v727: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s; 27440/329610 objects misplaced (8.325%); 47 MiB/s, 751 objects/s recovering 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[29779]: osdmap e336: 8 total, 8 up, 6 in 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[38113]: osdmap e335: 8 total, 8 up, 6 in 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[38113]: pgmap v727: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s; 27440/329610 objects misplaced (8.325%); 47 MiB/s, 751 objects/s recovering 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:00.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:00 smithi038 ceph-mon[38113]: osdmap e336: 8 total, 8 up, 6 in 2024-03-04T16:19:00.829 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:00.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:00 smithi176 ceph-mon[33619]: osdmap e335: 8 total, 8 up, 6 in 2024-03-04T16:19:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:00 smithi176 ceph-mon[33619]: pgmap v727: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.35k op/s; 27440/329610 objects misplaced (8.325%); 47 MiB/s, 751 objects/s recovering 2024-03-04T16:19:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:00 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:00 smithi176 ceph-mon[33619]: osdmap e336: 8 total, 8 up, 6 in 2024-03-04T16:19:00.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:00.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:00.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:01.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:01.110 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:01.155 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:01.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:01.290 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:01.392 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:01.434 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:01.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:01.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:01.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:01.706 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:01.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:01.848 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:01.866 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:01.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:01.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:19:01.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:02.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:02.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:02.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:02.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:02.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:02.454 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:02.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:02.659 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:02.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:02 smithi038 ceph-mon[29779]: pgmap v729: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s; 27440/329610 objects misplaced (8.325%); 23 MiB/s, 373 objects/s recovering 2024-03-04T16:19:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:02 smithi038 ceph-mon[38113]: pgmap v729: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s; 27440/329610 objects misplaced (8.325%); 23 MiB/s, 373 objects/s recovering 2024-03-04T16:19:02.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.807 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:02.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:02.959 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:02.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:03.061 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:03.093 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:02 smithi176 ceph-mon[33619]: pgmap v729: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 150 active+clean; 6.7 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 955 op/s; 27440/329610 objects misplaced (8.325%); 23 MiB/s, 373 objects/s recovering 2024-03-04T16:19:03.194 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:03.328 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:03.360 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:03.462 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.495 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:03.531 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:03.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:03.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:03.650 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:19:03.471+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e336 definitely_dead 0 2024-03-04T16:19:03.651 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:03.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:03.738 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.2. 2024-03-04T16:19:03.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:03.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:03.916 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:03.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:03.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/674199836' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[29779]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/674199836' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:03 smithi038 ceph-mon[38113]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:04.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:04.044 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:04.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:03 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/674199836' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:03 smithi176 ceph-mon[33619]: from='client.35717 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-03-04T16:19:04.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:04.171 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:04.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:04.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:04.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:04.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:04.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:04.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:04.553 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:04.655 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:04.684 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:04.785 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:04.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:04.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:04.928 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[29779]: pgmap v730: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.22k op/s; 25484/320493 objects misplaced (7.951%); 31 MiB/s, 491 objects/s recovering 2024-03-04T16:19:04.928 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[29779]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-04T16:19:04.929 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[29779]: osdmap e337: 8 total, 8 up, 7 in 2024-03-04T16:19:04.929 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[38113]: pgmap v730: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.22k op/s; 25484/320493 objects misplaced (7.951%); 31 MiB/s, 491 objects/s recovering 2024-03-04T16:19:04.929 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[38113]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-04T16:19:04.929 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:04 smithi038 ceph-mon[38113]: osdmap e337: 8 total, 8 up, 7 in 2024-03-04T16:19:04.938 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:04.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:05.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:05.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:04 smithi176 ceph-mon[33619]: pgmap v730: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 6.5 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.22k op/s; 25484/320493 objects misplaced (7.951%); 31 MiB/s, 491 objects/s recovering 2024-03-04T16:19:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:04 smithi176 ceph-mon[33619]: from='client.35717 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-03-04T16:19:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:04 smithi176 ceph-mon[33619]: osdmap e337: 8 total, 8 up, 7 in 2024-03-04T16:19:05.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:05.206 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:05.258 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:19:05.294 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:19:05.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:05.330 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:19:05.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:05.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:05.370 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:19:05.414 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:19:05.443 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:05.448 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:19:05.471 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:05.484 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:19:05.522 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:19:05.525 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:19:05.526 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:19:05.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:05.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:05.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:05.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:05.736 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:05.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:05.882 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:05.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:05.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: osdmap e338: 8 total, 8 up, 7 in 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.18 continuing backfill to osd.1 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.18 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9257 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.14 continuing backfill to osd.1 from (309'3500,336'8966] 10:2b43347c:::benchmark_data_smithi038_155391_object102242:head to 336'8966 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 336'8966 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.13 continuing backfill to osd.1 from (309'3360,336'9051] 10:c8004ac0:::benchmark_data_smithi038_155391_object31610:head to 336'9051 2024-03-04T16:19:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.13 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9051 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9023 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.12 continuing backfill to osd.5 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.12 continuing backfill to osd.7 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.18 continuing backfill to osd.3 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/990715695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 336'10584 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[29779]: osdmap e339: 8 total, 8 up, 7 in 2024-03-04T16:19:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: osdmap e338: 8 total, 8 up, 7 in 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.18 continuing backfill to osd.1 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.18 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9257 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.14 continuing backfill to osd.1 from (309'3500,336'8966] 10:2b43347c:::benchmark_data_smithi038_155391_object102242:head to 336'8966 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 336'8966 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.13 continuing backfill to osd.1 from (309'3360,336'9051] 10:c8004ac0:::benchmark_data_smithi038_155391_object31610:head to 336'9051 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.13 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9051 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9023 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.12 continuing backfill to osd.5 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.12 continuing backfill to osd.7 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:05.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.18 continuing backfill to osd.3 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:05.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/990715695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:05.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:05.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 336'10584 2024-03-04T16:19:05.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:05 smithi038 ceph-mon[38113]: osdmap e339: 8 total, 8 up, 7 in 2024-03-04T16:19:06.016 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: osdmap e338: 8 total, 8 up, 7 in 2024-03-04T16:19:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.18 continuing backfill to osd.1 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.18 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9257 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.14 continuing backfill to osd.1 from (309'3500,336'8966] 10:2b43347c:::benchmark_data_smithi038_155391_object102242:head to 336'8966 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 336'8966 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.13 continuing backfill to osd.1 from (309'3360,336'9051] 10:c8004ac0:::benchmark_data_smithi038_155391_object31610:head to 336'9051 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.13 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9051 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9023 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.12 continuing backfill to osd.5 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.12 continuing backfill to osd.7 from (309'3700,336'9023] 10:4eb6fdd8:::benchmark_data_smithi038_155391_object51004:head to 336'9023 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.18 continuing backfill to osd.3 from (309'3712,336'9257] 10:1f75061e:::benchmark_data_smithi038_155391_object52859:head to 336'9257 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/990715695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 336'10584 2024-03-04T16:19:06.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:05 smithi176 ceph-mon[33619]: osdmap e339: 8 total, 8 up, 7 in 2024-03-04T16:19:06.119 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:06.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:06.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:06.288 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:06.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:06.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:06.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:06.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:06.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:06.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:06.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:06.957 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:06.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:07.059 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:07.098 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9101 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 336'10436 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9204 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9046 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9005 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9008 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 336'10619 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:19:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:06 smithi176 ceph-mon[33619]: pgmap v734: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 965 B/s rd, 0 B/s wr, 1.13k op/s; 15310/311691 objects misplaced (4.912%); 34 MiB/s, 551 objects/s recovering 2024-03-04T16:19:07.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:07.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9101 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 336'10436 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9204 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9046 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9005 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9008 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 336'10619 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:19:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[29779]: pgmap v734: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 965 B/s rd, 0 B/s wr, 1.13k op/s; 15310/311691 objects misplaced (4.912%); 34 MiB/s, 551 objects/s recovering 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9101 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 336'10436 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9204 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9046 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9005 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 336'9008 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 336'10619 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:19:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:06 smithi038 ceph-mon[38113]: pgmap v734: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 965 B/s rd, 0 B/s wr, 1.13k op/s; 15310/311691 objects misplaced (4.912%); 34 MiB/s, 551 objects/s recovering 2024-03-04T16:19:07.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.367 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:07.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:07.468 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:07.497 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:07.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:07.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.641 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:07.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:07.771 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:07.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:07.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:07.903 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:07.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:08.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.046 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:08.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:07 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:19:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:07 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:19:08.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:08.181 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:08.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:07 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:19:08.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:07 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:19:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:07 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:19:08.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:07 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:19:08.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:08.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:08.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:08.458 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:08.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:08.585 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:08.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.727 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:08.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:08.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:08.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:08.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:08.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:09.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:08 smithi176 ceph-mon[33619]: pgmap v735: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 996 op/s; 15310/311691 objects misplaced (4.912%); 30 MiB/s, 487 objects/s recovering 2024-03-04T16:19:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:09.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:09.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:09.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:09.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:08 smithi038 ceph-mon[29779]: pgmap v735: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 996 op/s; 15310/311691 objects misplaced (4.912%); 30 MiB/s, 487 objects/s recovering 2024-03-04T16:19:09.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:08 smithi038 ceph-mon[38113]: pgmap v735: 158 pgs: 1 remapped+peering, 10 peering, 1 active+remapped+backfill_toofull, 1 unknown, 5 active+remapped+backfilling, 140 active+clean; 6.3 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 996 op/s; 15310/311691 objects misplaced (4.912%); 30 MiB/s, 487 objects/s recovering 2024-03-04T16:19:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:09.278 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:09.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:09.404 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:09.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:09.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:09.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:09.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.816 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:09.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:09.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:09.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:09.958 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:09.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:19:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:19:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:19:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:19:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[29779]: osdmap e340: 8 total, 8 up, 7 in 2024-03-04T16:19:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:19:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:19:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:19:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:19:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:09 smithi038 ceph-mon[38113]: osdmap e340: 8 total, 8 up, 7 in 2024-03-04T16:19:10.060 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:10.106 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:19:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:19:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:09 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:19:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:09 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:19:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:09 smithi176 ceph-mon[33619]: osdmap e340: 8 total, 8 up, 7 in 2024-03-04T16:19:10.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:10.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:10.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:10.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:10.383 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:10.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:10.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:10.530 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:10.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:10.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:10.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:10.798 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:10.900 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:10.932 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:10.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:11.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:11.081 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:10 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:19:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:10 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:19:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:10 smithi176 ceph-mon[33619]: pgmap v737: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 848 op/s; 63638/301428 objects misplaced (21.112%); 41 MiB/s, 665 objects/s recovering 2024-03-04T16:19:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:10 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:10 smithi176 ceph-mon[33619]: osdmap e341: 8 total, 8 up, 7 in 2024-03-04T16:19:11.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:11.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:11.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[29779]: pgmap v737: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 848 op/s; 63638/301428 objects misplaced (21.112%); 41 MiB/s, 665 objects/s recovering 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[29779]: osdmap e341: 8 total, 8 up, 7 in 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[38113]: pgmap v737: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 864 B/s rd, 0 B/s wr, 848 op/s; 63638/301428 objects misplaced (21.112%); 41 MiB/s, 665 objects/s recovering 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:11.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:10 smithi038 ceph-mon[38113]: osdmap e341: 8 total, 8 up, 7 in 2024-03-04T16:19:11.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:11.371 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:11.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:11.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:11.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:11.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:11.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:11.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:11.888 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:11.918 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:11.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:12.020 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:12.051 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:12.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:12.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:12.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:12.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:12.332 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:12.412 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-03-04T16:19:12.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:12.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:12.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.659 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:12.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:12.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:12.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:12.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:12.943 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:12.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:13.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:13.075 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:13.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:13.212 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[29779]: pgmap v739: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 813 B/s rd, 0 B/s wr, 544 op/s; 63638/301428 objects misplaced (21.112%); 38 MiB/s, 622 objects/s recovering 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[29779]: osdmap e342: 8 total, 8 up, 7 in 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[38113]: pgmap v739: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 813 B/s rd, 0 B/s wr, 544 op/s; 63638/301428 objects misplaced (21.112%); 38 MiB/s, 622 objects/s recovering 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:19:13.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:12 smithi038 ceph-mon[38113]: osdmap e342: 8 total, 8 up, 7 in 2024-03-04T16:19:13.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:12 smithi176 ceph-mon[33619]: pgmap v739: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.1 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 813 B/s rd, 0 B/s wr, 544 op/s; 63638/301428 objects misplaced (21.112%); 38 MiB/s, 622 objects/s recovering 2024-03-04T16:19:13.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:12 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:19:13.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:12 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:19:13.254 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:12 smithi176 ceph-mon[33619]: osdmap e342: 8 total, 8 up, 7 in 2024-03-04T16:19:13.314 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:13.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:13.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:13.449 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:13.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:13.578 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:13.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:13.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:13.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:13.692 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:19:13.692 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:19:13.692 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-04T16:19:13.692 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:19:13.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:13.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:13.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:13.909 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:13.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:14.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:14.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:14.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:14.181 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:14.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:14.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:14.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:14.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:14.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:14.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:14.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:14.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:14.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:14.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:14.888 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:14.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:14.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:19:14.898 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:19:14.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:15.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:15.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:15.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:15.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:14 smithi038 ceph-mon[38113]: pgmap v741: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 927 op/s; 60819/295029 objects misplaced (20.615%); 61 MiB/s, 987 objects/s recovering 2024-03-04T16:19:15.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:19:15.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:14 smithi038 ceph-mon[29779]: pgmap v741: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 927 op/s; 60819/295029 objects misplaced (20.615%); 61 MiB/s, 987 objects/s recovering 2024-03-04T16:19:15.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:19:15.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:15.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:15.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:19:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:14 smithi176 ceph-mon[33619]: pgmap v741: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 927 op/s; 60819/295029 objects misplaced (20.615%); 61 MiB/s, 987 objects/s recovering 2024-03-04T16:19:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:19:15.412 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:15.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:15.544 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.576 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:15.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:15.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:15.708 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:15.810 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:15.833 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:15.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:15.971 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:15.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:16.073 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:16.100 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:16.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:16.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:16.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:19:16.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:19:16.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[29779]: osdmap e343: 8 total, 8 up, 7 in 2024-03-04T16:19:16.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[29779]: osdmap e344: 8 total, 8 up, 7 in 2024-03-04T16:19:16.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:19:16.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:19:16.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[38113]: osdmap e343: 8 total, 8 up, 7 in 2024-03-04T16:19:16.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:15 smithi038 ceph-mon[38113]: osdmap e344: 8 total, 8 up, 7 in 2024-03-04T16:19:16.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:15 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:19:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1537402028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:19:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:15 smithi176 ceph-mon[33619]: osdmap e343: 8 total, 8 up, 7 in 2024-03-04T16:19:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:15 smithi176 ceph-mon[33619]: osdmap e344: 8 total, 8 up, 7 in 2024-03-04T16:19:16.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:16.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:16.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:16.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:16.637 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:16.739 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:16.763 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:16.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:16.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:16.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:16.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:17.025 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:17.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:17.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:17.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[29779]: pgmap v744: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 966 B/s rd, 0 B/s wr, 953 op/s; 57756/286293 objects misplaced (20.174%); 33 MiB/s, 525 objects/s recovering 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[29779]: osdmap e345: 8 total, 8 up, 7 in 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[38113]: pgmap v744: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 966 B/s rd, 0 B/s wr, 953 op/s; 57756/286293 objects misplaced (20.174%); 33 MiB/s, 525 objects/s recovering 2024-03-04T16:19:17.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:16 smithi038 ceph-mon[38113]: osdmap e345: 8 total, 8 up, 7 in 2024-03-04T16:19:17.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:17.276 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:16 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:19:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:16 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:19:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:16 smithi176 ceph-mon[33619]: pgmap v744: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 966 B/s rd, 0 B/s wr, 953 op/s; 57756/286293 objects misplaced (20.174%); 33 MiB/s, 525 objects/s recovering 2024-03-04T16:19:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:16 smithi176 ceph-mon[33619]: osdmap e345: 8 total, 8 up, 7 in 2024-03-04T16:19:17.378 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:17.408 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:17.414 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 2] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:19:17.414 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:19:17.414 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-04T16:19:17.414 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-04T16:19:17.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:19:17.510 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:17.561 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:17.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:17.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:17.828 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:17.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:17.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:17.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:18.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:18.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:18.130 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:18.144 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:18.157 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:18.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:18.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:18.305 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:18.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:18.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:18.610 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[29779]: osdmap e346: 8 total, 8 up, 7 in 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[38113]: osdmap e346: 8 total, 8 up, 7 in 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:19:18.653 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:18 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:19:18.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:18.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:18.855 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:18 smithi176 ceph-mon[33619]: osdmap e346: 8 total, 8 up, 7 in 2024-03-04T16:19:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:18 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:19:18.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:18 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:19:18.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:18.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:18.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:19.027 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:19.129 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:19.170 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:19.272 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:19.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:19.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:19.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:19.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:19 smithi038 ceph-mon[29779]: pgmap v747: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 57756/286293 objects misplaced (20.174%); 12 MiB/s, 192 objects/s recovering 2024-03-04T16:19:19.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:19 smithi038 ceph-mon[38113]: pgmap v747: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 57756/286293 objects misplaced (20.174%); 12 MiB/s, 192 objects/s recovering 2024-03-04T16:19:19.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:19.564 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:19.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:19.703 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:19.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:19.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:19.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:19 smithi176 ceph-mon[33619]: pgmap v747: 158 pgs: 7 active+remapped+backfill_wait, 143 active+clean, 5 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 727 op/s; 57756/286293 objects misplaced (20.174%); 12 MiB/s, 192 objects/s recovering 2024-03-04T16:19:19.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:19.973 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:19.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:19.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:20.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:19:20.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:20.133 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:20.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:20.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.268 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:20.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:20.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:20.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:20.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:20.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.546 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:20.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:20.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:20.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:20.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:20.778 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:20.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:20.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:20.950 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:20.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:21.052 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:21.084 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:21.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:21.217 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:21.319 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:21.342 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:21.354 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:19:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=sqlstore.transactions t=2024-03-04T16:19:21.061307654Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-04T16:19:21.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:21.487 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:21.589 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:21.607 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:21 smithi176 ceph-mon[33619]: pgmap v748: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 838 op/s; 51663/274332 objects misplaced (18.832%); 45 MiB/s, 722 objects/s recovering 2024-03-04T16:19:21.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:21 smithi176 ceph-mon[33619]: osdmap e347: 8 total, 8 up, 7 in 2024-03-04T16:19:21.612 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:21.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:21.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:21.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:21 smithi038 ceph-mon[29779]: pgmap v748: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 838 op/s; 51663/274332 objects misplaced (18.832%); 45 MiB/s, 722 objects/s recovering 2024-03-04T16:19:21.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:21 smithi038 ceph-mon[29779]: osdmap e347: 8 total, 8 up, 7 in 2024-03-04T16:19:21.743 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:21 smithi038 ceph-mon[38113]: pgmap v748: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 838 op/s; 51663/274332 objects misplaced (18.832%); 45 MiB/s, 722 objects/s recovering 2024-03-04T16:19:21.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:21 smithi038 ceph-mon[38113]: osdmap e347: 8 total, 8 up, 7 in 2024-03-04T16:19:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:21.845 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:21.876 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:21.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:21.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:21.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:22.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:22.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:22.135 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:22.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:22.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.262 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:22.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:22.388 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:22.490 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:22.513 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:22.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:22.647 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: 10.6 scrub starts 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: osdmap e348: 8 total, 8 up, 7 in 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: pgmap v751: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 754 op/s; 51663/274332 objects misplaced (18.832%); 41 MiB/s, 649 objects/s recovering 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[29779]: 10.6 scrub ok 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: 10.6 scrub starts 2024-03-04T16:19:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: osdmap e348: 8 total, 8 up, 7 in 2024-03-04T16:19:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:19:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: pgmap v751: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 754 op/s; 51663/274332 objects misplaced (18.832%); 41 MiB/s, 649 objects/s recovering 2024-03-04T16:19:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:19:22.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:22 smithi038 ceph-mon[38113]: 10.6 scrub ok 2024-03-04T16:19:22.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:22.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: 10.6 scrub starts 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: osdmap e348: 8 total, 8 up, 7 in 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: pgmap v751: 158 pgs: 7 active+remapped+backfill_wait, 145 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 754 op/s; 51663/274332 objects misplaced (18.832%); 41 MiB/s, 649 objects/s recovering 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:19:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:22 smithi176 ceph-mon[33619]: 10.6 scrub ok 2024-03-04T16:19:22.881 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:22.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:22.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:23.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:23.050 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:23.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:23.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.186 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:23.287 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:23.310 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:23.411 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:23.445 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:23.547 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:23.579 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:23.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: 10.11 deep-scrub starts 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: osdmap e349: 8 total, 8 up, 7 in 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:23.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:23.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: 10.11 deep-scrub starts 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: osdmap e349: 8 total, 8 up, 7 in 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: 10.11 deep-scrub starts 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: osdmap e349: 8 total, 8 up, 7 in 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:23.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:23.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:23.837 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:23.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:23.973 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:23.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:24.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:24.106 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:24.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.241 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:24.342 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:24.370 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:24.472 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:24.496 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:24.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:24.626 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:24.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:24.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:24 smithi038 ceph-mon[29779]: pgmap v753: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 50136/269205 objects misplaced (18.624%); 46 MiB/s, 728 objects/s recovering 2024-03-04T16:19:24.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:24 smithi038 ceph-mon[38113]: pgmap v753: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 50136/269205 objects misplaced (18.624%); 46 MiB/s, 728 objects/s recovering 2024-03-04T16:19:24.779 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:24.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:24 smithi176 ceph-mon[33619]: pgmap v753: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.5 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 50136/269205 objects misplaced (18.624%); 46 MiB/s, 728 objects/s recovering 2024-03-04T16:19:24.881 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:24.912 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:24.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:25.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:25.046 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:25.148 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:25.190 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:25.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:25.327 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:25.428 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:25.461 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:25.563 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.595 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:25.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:25.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:25.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:25.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:25.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:19:25.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:19:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:19:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:25 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:19:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:25.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:25.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:25 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:25 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:19:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:25 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:19:25.894 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:25.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:25.995 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:26.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:26.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:26.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:26.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:26.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:26.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:26.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:26.505 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:26.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.641 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:26.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:26.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:26 smithi038 ceph-mon[29779]: osdmap e350: 8 total, 8 up, 7 in 2024-03-04T16:19:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:26 smithi038 ceph-mon[29779]: pgmap v755: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 967 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 23 MiB/s, 365 objects/s recovering 2024-03-04T16:19:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:26 smithi038 ceph-mon[38113]: osdmap e350: 8 total, 8 up, 7 in 2024-03-04T16:19:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:26 smithi038 ceph-mon[38113]: pgmap v755: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 967 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 23 MiB/s, 365 objects/s recovering 2024-03-04T16:19:26.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:26 smithi176 ceph-mon[33619]: osdmap e350: 8 total, 8 up, 7 in 2024-03-04T16:19:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:26 smithi176 ceph-mon[33619]: pgmap v755: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 967 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 23 MiB/s, 365 objects/s recovering 2024-03-04T16:19:26.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:26.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:26.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:27.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:27.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:27.169 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.204 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:27.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:27.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:27.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:27.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:27.462 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.505 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:27.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:27.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.684 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:27.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:27.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:27 smithi038 ceph-mon[29779]: 10.c scrub starts 2024-03-04T16:19:27.734 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:27 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:19:27.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:27 smithi038 ceph-mon[38113]: 10.c scrub starts 2024-03-04T16:19:27.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:27 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:19:27.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:27.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:27.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:27 smithi176 ceph-mon[33619]: 10.c scrub starts 2024-03-04T16:19:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:27 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:19:27.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:27.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:28.072 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:28.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:28.143 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:28.170 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:19:28.170 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:19:28.170 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:19:28.170 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:19:28.170 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:19:28.245 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:28.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:28.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:28.436 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:28.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:28.538 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:28.578 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:28.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.724 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:28.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:28.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:28.867 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:28.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:28.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:19:28.969 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:29.000 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:29.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:29.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:29.178 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:29.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[29779]: pgmap v756: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:19:29.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[29779]: 10.c scrub ok 2024-03-04T16:19:29.204 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[38113]: pgmap v756: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:19:29.204 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.204 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.204 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:29 smithi038 ceph-mon[38113]: 10.c scrub ok 2024-03-04T16:19:29.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:29.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.331 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:29.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:29 smithi176 ceph-mon[33619]: pgmap v756: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 816 B/s rd, 0 op/s; 47086/262701 objects misplaced (17.924%); 19 MiB/s, 308 objects/s recovering 2024-03-04T16:19:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:29 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:29 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:29 smithi176 ceph-mon[33619]: 10.c scrub ok 2024-03-04T16:19:29.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:29.539 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:29.599 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:19:29.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:29.646 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:19:29.690 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:19:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.710 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:29.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:29.743 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:19:29.785 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:19:29.811 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:29.829 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:19:29.841 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:29.873 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:19:29.921 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:19:29.929 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:19:29.943 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:29.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:29.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:30.075 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:30.123 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:30.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:30.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:30.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:30 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:30 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4157318933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:30.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:30.413 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:30.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4157318933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:30.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4157318933' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:30.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:30.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:30.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:30.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:30.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:30.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:30.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:30.969 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:30.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:31.070 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:31.102 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:31.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:31.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:31.348 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:31.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:31.482 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.521 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:31.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:31.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:31.661 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:31.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[29779]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 306 op/s; 42092/249759 objects misplaced (16.853%); 31 MiB/s, 495 objects/s recovering 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[29779]: osdmap e351: 8 total, 8 up, 7 in 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[29779]: 3.11 deep-scrub starts 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[29779]: 3.11 deep-scrub ok 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[38113]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 306 op/s; 42092/249759 objects misplaced (16.853%); 31 MiB/s, 495 objects/s recovering 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[38113]: osdmap e351: 8 total, 8 up, 7 in 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[38113]: 3.11 deep-scrub starts 2024-03-04T16:19:31.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:31 smithi038 ceph-mon[38113]: 3.11 deep-scrub ok 2024-03-04T16:19:31.763 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:31.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:31 smithi176 ceph-mon[33619]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 306 op/s; 42092/249759 objects misplaced (16.853%); 31 MiB/s, 495 objects/s recovering 2024-03-04T16:19:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:31 smithi176 ceph-mon[33619]: osdmap e351: 8 total, 8 up, 7 in 2024-03-04T16:19:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:31 smithi176 ceph-mon[33619]: 3.11 deep-scrub starts 2024-03-04T16:19:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:31 smithi176 ceph-mon[33619]: 3.11 deep-scrub ok 2024-03-04T16:19:31.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:31.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:31.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:32.049 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:32.076 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:32.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:32.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:32.210 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:32.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:32.344 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:32.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:32.487 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:32.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:32.612 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:32.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:32.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:32.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:32.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:32.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:32.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:32.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:33.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:33.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.161 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:33.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:33.293 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:33.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:33.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:33.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:33.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:33 smithi176 ceph-mon[33619]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 810 op/s; 42092/249759 objects misplaced (16.853%); 24 MiB/s, 379 objects/s recovering 2024-03-04T16:19:33.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:33.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:33.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:33.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:33.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:33 smithi038 ceph-mon[29779]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 810 op/s; 42092/249759 objects misplaced (16.853%); 24 MiB/s, 379 objects/s recovering 2024-03-04T16:19:33.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:33.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:33 smithi038 ceph-mon[38113]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 144 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 810 op/s; 42092/249759 objects misplaced (16.853%); 24 MiB/s, 379 objects/s recovering 2024-03-04T16:19:33.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:33.837 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:33.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:33.939 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:33.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:33.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:34.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:34.111 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:34.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:34.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:34.249 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:34.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:34.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:34.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:34.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:34.537 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:34.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:34.671 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:34.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:34.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:34.773 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:34.806 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:34.907 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:34.940 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:34.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:34.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:35.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.116 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:35.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:35.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:35.261 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:35.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:35.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:35.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: pgmap v760: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 40315/244407 objects misplaced (16.495%); 24 MiB/s, 381 objects/s recovering 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: pgmap v760: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 40315/244407 objects misplaced (16.495%); 24 MiB/s, 381 objects/s recovering 2024-03-04T16:19:35.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:19:35.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:19:35.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:19:35.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:19:35.494 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:35 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:35.516 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:35.556 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:35.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:35.693 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:35.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.854 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:35.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: pgmap v760: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 5.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 742 op/s; 40315/244407 objects misplaced (16.495%); 24 MiB/s, 381 objects/s recovering 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:19:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:35 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:35.955 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:35.988 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:35.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:36.089 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.124 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:36.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:36.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:36.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:36.278 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:36.374 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:36.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:36.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:36 smithi038 ceph-mon[29779]: pgmap v761: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 37520/237222 objects misplaced (15.816%); 24 MiB/s, 391 objects/s recovering 2024-03-04T16:19:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:36 smithi038 ceph-mon[38113]: pgmap v761: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 37520/237222 objects misplaced (15.816%); 24 MiB/s, 391 objects/s recovering 2024-03-04T16:19:36.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:36.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:36.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:36.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:36.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:36.817 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:36 smithi176 ceph-mon[33619]: pgmap v761: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 850 op/s; 37520/237222 objects misplaced (15.816%); 24 MiB/s, 391 objects/s recovering 2024-03-04T16:19:36.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:36.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:36.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:37.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:37.091 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:37.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:37.224 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:37.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:37.358 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:19:37.405 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:19:37.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:37.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:37.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:37.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:37.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:37.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:37 smithi038 ceph-mon[29779]: osdmap e352: 8 total, 8 up, 7 in 2024-03-04T16:19:37.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:37 smithi038 ceph-mon[29779]: 10.11 deep-scrub ok 2024-03-04T16:19:37.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:37 smithi038 ceph-mon[38113]: osdmap e352: 8 total, 8 up, 7 in 2024-03-04T16:19:37.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:37 smithi038 ceph-mon[38113]: 10.11 deep-scrub ok 2024-03-04T16:19:37.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:37.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:19:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:37 smithi176 ceph-mon[33619]: osdmap e352: 8 total, 8 up, 7 in 2024-03-04T16:19:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:37 smithi176 ceph-mon[33619]: 10.11 deep-scrub ok 2024-03-04T16:19:37.876 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:37.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:37.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:38.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:38.041 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:38.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:38.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:38.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:38.313 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:38.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:38.398 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-03-04T16:19:36.413881+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5169715574,"num_objects":79074,"num_object_clones":0,"num_object_copies":237222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":37520,"num_objects_unfound":0,"num_objects_dirty":79074,"num_whiteouts":0,"num_read":2351,"num_read_kb":2197,"num_write":188185,"num_write_kb":8430590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38108,"num_bytes_recovered":2480419632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140898,"ondisk_log_size":140898,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":500,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22694312,"kb_used_data":17405740,"kb_used_omap":10,"kb_used_meta":5288245,"kb_avail":633521752,"statfs":{"total":671965249536,"available":648726274048,"internally_reserved":0,"allocated":17823477760,"data_stored":17819627982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":5415162988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556597248,"num_objects":-8493,"num_object_clones":0,"num_object_copies":-25479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9566,"num_objects_unfound":0,"num_objects_dirty":-8493,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8493,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3913,"num_bytes_recovered":256442368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.755956"},"pg_stats":[{"pgid":"10.16","version":"351'9766","reported_seq":16180,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.065905+0000","last_change":"2024-03-04T16:19:16.373647+0000","last_active":"2024-03-04T16:19:33.065905+0000","last_peered":"2024-03-04T16:19:33.065905+0000","last_clean":"2024-03-04T16:19:33.065905+0000","last_became_active":"2024-03-04T16:19:16.362564+0000","last_became_peered":"2024-03-04T16:19:16.362564+0000","last_unstale":"2024-03-04T16:19:33.065905+0000","last_undegraded":"2024-03-04T16:19:33.065905+0000","last_fullsized":"2024-03-04T16:19:33.065905+0000","mapping_epoch":344,"log_start":"307'2900","ondisk_log_start":"307'2900","created":320,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6866,"log_dups_size":0,"ondisk_log_size":6866,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:33.465638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162103296,"num_objects":2474,"num_object_clones":0,"num_object_copies":7422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6016,"num_write_kb":271648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3129,"num_bytes_recovered":205062144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":619,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366583+0000","last_change":"2024-03-04T16:16:48.504644+0000","last_active":"2024-03-04T16:19:30.366583+0000","last_peered":"2024-03-04T16:19:30.366583+0000","last_clean":"2024-03-04T16:19:30.366583+0000","last_became_active":"2024-03-04T16:02:49.786443+0000","last_became_peered":"2024-03-04T16:02:49.786443+0000","last_unstale":"2024-03-04T16:19:30.366583+0000","last_undegraded":"2024-03-04T16:19:30.366583+0000","last_fullsized":"2024-03-04T16:19:30.366583+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:48.504543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:16:48.504543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":623,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.379363+0000","last_change":"2024-03-04T16:17:48.323042+0000","last_active":"2024-03-04T16:19:30.379363+0000","last_peered":"2024-03-04T16:19:30.379363+0000","last_clean":"2024-03-04T16:19:30.379363+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:19:30.379363+0000","last_undegraded":"2024-03-04T16:19:30.379363+0000","last_fullsized":"2024-03-04T16:19:30.379363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.322908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.322908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":494,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390662+0000","last_change":"2024-03-04T16:19:04.848699+0000","last_active":"2024-03-04T16:19:30.390662+0000","last_peered":"2024-03-04T16:19:30.390662+0000","last_clean":"2024-03-04T16:19:30.390662+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:19:30.390662+0000","last_undegraded":"2024-03-04T16:19:30.390662+0000","last_fullsized":"2024-03-04T16:19:30.390662+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.623578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.623578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":668,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.365953+0000","last_change":"2024-03-04T16:17:07.218989+0000","last_active":"2024-03-04T16:19:30.365953+0000","last_peered":"2024-03-04T16:19:30.365953+0000","last_clean":"2024-03-04T16:19:30.365953+0000","last_became_active":"2024-03-04T16:10:29.253041+0000","last_became_peered":"2024-03-04T16:10:29.253041+0000","last_unstale":"2024-03-04T16:19:30.365953+0000","last_undegraded":"2024-03-04T16:19:30.365953+0000","last_fullsized":"2024-03-04T16:19:30.365953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:17:07.218848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.218848+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.17","version":"351'9686","reported_seq":17801,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:32.808382+0000","last_change":"2024-03-04T16:18:51.357144+0000","last_active":"2024-03-04T16:19:32.808382+0000","last_peered":"2024-03-04T16:19:32.808382+0000","last_clean":"2024-03-04T16:19:32.808382+0000","last_became_active":"2024-03-04T16:18:51.350040+0000","last_became_peered":"2024-03-04T16:18:51.350040+0000","last_unstale":"2024-03-04T16:19:32.808382+0000","last_undegraded":"2024-03-04T16:19:32.808382+0000","last_fullsized":"2024-03-04T16:19:32.808382+0000","mapping_epoch":329,"log_start":"312'4649","ondisk_log_start":"312'4649","created":320,"last_epoch_clean":330,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5037,"log_dups_size":0,"ondisk_log_size":5037,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:33.798953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164233216,"num_objects":2506,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5970,"num_write_kb":271232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3988,"num_bytes_recovered":261357568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":617,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.385361+0000","last_change":"2024-03-04T16:16:55.868681+0000","last_active":"2024-03-04T16:19:30.385361+0000","last_peered":"2024-03-04T16:19:30.385361+0000","last_clean":"2024-03-04T16:19:30.385361+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:19:30.385361+0000","last_undegraded":"2024-03-04T16:19:30.385361+0000","last_fullsized":"2024-03-04T16:19:30.385361+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.868581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.868581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366152+0000","last_change":"2024-03-04T16:19:04.758811+0000","last_active":"2024-03-04T16:19:30.366152+0000","last_peered":"2024-03-04T16:19:30.366152+0000","last_clean":"2024-03-04T16:19:30.366152+0000","last_became_active":"2024-03-04T16:19:04.758455+0000","last_became_peered":"2024-03-04T16:19:04.758455+0000","last_unstale":"2024-03-04T16:19:30.366152+0000","last_undegraded":"2024-03-04T16:19:30.366152+0000","last_fullsized":"2024-03-04T16:19:30.366152+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.629592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.382583+0000","last_change":"2024-03-04T16:19:04.779817+0000","last_active":"2024-03-04T16:19:30.382583+0000","last_peered":"2024-03-04T16:19:30.382583+0000","last_clean":"2024-03-04T16:19:30.382583+0000","last_became_active":"2024-03-04T16:19:04.779498+0000","last_became_peered":"2024-03-04T16:19:04.779498+0000","last_unstale":"2024-03-04T16:19:30.382583+0000","last_undegraded":"2024-03-04T16:19:30.382583+0000","last_fullsized":"2024-03-04T16:19:30.382583+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.030580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.030580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":750,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381995+0000","last_change":"2024-03-04T16:19:04.864691+0000","last_active":"2024-03-04T16:19:30.381995+0000","last_peered":"2024-03-04T16:19:30.381995+0000","last_clean":"2024-03-04T16:19:30.381995+0000","last_became_active":"2024-03-04T16:19:04.778671+0000","last_became_peered":"2024-03-04T16:19:04.778671+0000","last_unstale":"2024-03-04T16:19:30.381995+0000","last_undegraded":"2024-03-04T16:19:30.381995+0000","last_fullsized":"2024-03-04T16:19:30.381995+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:15:51.042010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.042010+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"351'9707","reported_seq":19533,"reported_epoch":351,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:19:33.874304+0000","last_change":"2024-03-04T16:19:05.019964+0000","last_active":"2024-03-04T16:19:33.874304+0000","last_peered":"2024-03-04T16:19:33.874304+0000","last_clean":"2024-03-04T16:18:20.339712+0000","last_became_active":"2024-03-04T16:19:04.893050+0000","last_became_peered":"2024-03-04T16:19:04.893050+0000","last_unstale":"2024-03-04T16:19:33.874304+0000","last_undegraded":"2024-03-04T16:19:33.874304+0000","last_fullsized":"2024-03-04T16:19:33.874304+0000","mapping_epoch":337,"log_start":"309'3500","ondisk_log_start":"309'3500","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162791424,"num_objects":2484,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":266,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5968,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4127,"num_bytes_recovered":270467072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":2484}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":483,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381286+0000","last_change":"2024-03-04T16:18:12.891866+0000","last_active":"2024-03-04T16:19:30.381286+0000","last_peered":"2024-03-04T16:19:30.381286+0000","last_clean":"2024-03-04T16:19:30.381286+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:19:30.381286+0000","last_undegraded":"2024-03-04T16:19:30.381286+0000","last_fullsized":"2024-03-04T16:19:30.381286+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:18:12.891748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:38.840961+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":622,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369823+0000","last_change":"2024-03-04T16:17:03.584844+0000","last_active":"2024-03-04T16:19:30.369823+0000","last_peered":"2024-03-04T16:19:30.369823+0000","last_clean":"2024-03-04T16:19:30.369823+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:19:30.369823+0000","last_undegraded":"2024-03-04T16:19:30.369823+0000","last_fullsized":"2024-03-04T16:19:30.369823+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.584766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.584766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":644,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370206+0000","last_change":"2024-03-04T16:16:18.732834+0000","last_active":"2024-03-04T16:19:30.370206+0000","last_peered":"2024-03-04T16:19:30.370206+0000","last_clean":"2024-03-04T16:19:30.370206+0000","last_became_active":"2024-03-04T16:10:29.367726+0000","last_became_peered":"2024-03-04T16:10:29.367726+0000","last_unstale":"2024-03-04T16:19:30.370206+0000","last_undegraded":"2024-03-04T16:19:30.370206+0000","last_fullsized":"2024-03-04T16:19:30.370206+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.732730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.732730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":688,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366883+0000","last_change":"2024-03-04T16:17:49.277087+0000","last_active":"2024-03-04T16:19:30.366883+0000","last_peered":"2024-03-04T16:19:30.366883+0000","last_clean":"2024-03-04T16:19:30.366883+0000","last_became_active":"2024-03-04T16:07:36.244533+0000","last_became_peered":"2024-03-04T16:07:36.244533+0000","last_unstale":"2024-03-04T16:19:30.366883+0000","last_undegraded":"2024-03-04T16:19:30.366883+0000","last_fullsized":"2024-03-04T16:19:30.366883+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:49.276987+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.276987+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"351'9861","reported_seq":21789,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.942647+0000","last_change":"2024-03-04T16:19:00.369215+0000","last_active":"2024-03-04T16:19:33.942647+0000","last_peered":"2024-03-04T16:19:33.942647+0000","last_clean":"2024-03-04T16:19:33.942647+0000","last_became_active":"2024-03-04T16:19:00.357537+0000","last_became_peered":"2024-03-04T16:19:00.357537+0000","last_unstale":"2024-03-04T16:19:33.942647+0000","last_undegraded":"2024-03-04T16:19:33.942647+0000","last_fullsized":"2024-03-04T16:19:33.942647+0000","mapping_epoch":335,"log_start":"312'4574","ondisk_log_start":"312'4574","created":320,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5287,"log_dups_size":0,"ondisk_log_size":5287,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160464896,"num_objects":2449,"num_object_clones":0,"num_object_copies":7347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6081,"num_write_kb":272928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3928,"num_bytes_recovered":257425408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":626,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.395314+0000","last_change":"2024-03-04T16:16:09.872300+0000","last_active":"2024-03-04T16:19:30.395314+0000","last_peered":"2024-03-04T16:19:30.395314+0000","last_clean":"2024-03-04T16:19:30.395314+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:19:30.395314+0000","last_undegraded":"2024-03-04T16:19:30.395314+0000","last_fullsized":"2024-03-04T16:19:30.395314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:16:09.872152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363810+0000","last_change":"2024-03-04T16:19:17.593219+0000","last_active":"2024-03-04T16:19:30.363810+0000","last_peered":"2024-03-04T16:19:30.363810+0000","last_clean":"2024-03-04T16:19:30.363810+0000","last_became_active":"2024-03-04T16:02:47.767076+0000","last_became_peered":"2024-03-04T16:02:47.767076+0000","last_unstale":"2024-03-04T16:19:30.363810+0000","last_undegraded":"2024-03-04T16:19:30.363810+0000","last_fullsized":"2024-03-04T16:19:30.363810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:17.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:19:17.593078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:40.856992+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":630,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368792+0000","last_change":"2024-03-04T16:16:06.843357+0000","last_active":"2024-03-04T16:19:30.368792+0000","last_peered":"2024-03-04T16:19:30.368792+0000","last_clean":"2024-03-04T16:19:30.368792+0000","last_became_active":"2024-03-04T16:02:43.710730+0000","last_became_peered":"2024-03-04T16:02:43.710730+0000","last_unstale":"2024-03-04T16:19:30.368792+0000","last_undegraded":"2024-03-04T16:19:30.368792+0000","last_fullsized":"2024-03-04T16:19:30.368792+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:06.843231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:06.843231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":666,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367202+0000","last_change":"2024-03-04T16:17:54.411929+0000","last_active":"2024-03-04T16:19:30.367202+0000","last_peered":"2024-03-04T16:19:30.367202+0000","last_clean":"2024-03-04T16:19:30.367202+0000","last_became_active":"2024-03-04T16:10:17.471785+0000","last_became_peered":"2024-03-04T16:10:17.471785+0000","last_unstale":"2024-03-04T16:19:30.367202+0000","last_undegraded":"2024-03-04T16:19:30.367202+0000","last_fullsized":"2024-03-04T16:19:30.367202+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-03-04T16:17:54.411724+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.411724+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.12","version":"351'9837","reported_seq":22816,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:33.862719+0000","last_change":"2024-03-04T16:19:05.509282+0000","last_active":"2024-03-04T16:19:33.862719+0000","last_peered":"2024-03-04T16:19:33.862719+0000","last_clean":"2024-03-04T16:18:20.339570+0000","last_became_active":"2024-03-04T16:19:04.891568+0000","last_became_peered":"2024-03-04T16:19:04.891568+0000","last_unstale":"2024-03-04T16:19:33.862719+0000","last_undegraded":"2024-03-04T16:19:33.862719+0000","last_fullsized":"2024-03-04T16:19:33.862719+0000","mapping_epoch":337,"log_start":"309'3700","ondisk_log_start":"309'3700","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6137,"log_dups_size":0,"ondisk_log_size":6137,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162955264,"num_objects":2487,"num_object_clones":0,"num_object_copies":7461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3365,"num_objects_unfound":0,"num_objects_dirty":2487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6071,"num_write_kb":273824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3254,"num_bytes_recovered":213254144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[0,1,4],"avail_no_missing":["0","1","3","4"],"object_location_counts":[{"shards":"0,1,4","objects":2487}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":627,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367439+0000","last_change":"2024-03-04T16:17:59.573768+0000","last_active":"2024-03-04T16:19:30.367439+0000","last_peered":"2024-03-04T16:19:30.367439+0000","last_clean":"2024-03-04T16:19:30.367439+0000","last_became_active":"2024-03-04T16:10:29.329680+0000","last_became_peered":"2024-03-04T16:10:29.329680+0000","last_unstale":"2024-03-04T16:19:30.367439+0000","last_undegraded":"2024-03-04T16:19:30.367439+0000","last_fullsized":"2024-03-04T16:19:30.367439+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:59.573389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:59.573389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.378150+0000","last_change":"2024-03-04T16:17:49.317278+0000","last_active":"2024-03-04T16:19:30.378150+0000","last_peered":"2024-03-04T16:19:30.378150+0000","last_clean":"2024-03-04T16:19:30.378150+0000","last_became_active":"2024-03-04T16:02:47.767619+0000","last_became_peered":"2024-03-04T16:02:47.767619+0000","last_unstale":"2024-03-04T16:19:30.378150+0000","last_undegraded":"2024-03-04T16:19:30.378150+0000","last_fullsized":"2024-03-04T16:19:30.378150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:49.317160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.317160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":475,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.380671+0000","last_change":"2024-03-04T16:18:04.653977+0000","last_active":"2024-03-04T16:19:30.380671+0000","last_peered":"2024-03-04T16:19:30.380671+0000","last_clean":"2024-03-04T16:19:30.380671+0000","last_became_active":"2024-03-04T16:07:36.212313+0000","last_became_peered":"2024-03-04T16:07:36.212313+0000","last_unstale":"2024-03-04T16:19:30.380671+0000","last_undegraded":"2024-03-04T16:19:30.380671+0000","last_fullsized":"2024-03-04T16:19:30.380671+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:04.653768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:04.653768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"286'34","reported_seq":53,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.365732+0000","last_change":"2024-03-04T16:19:05.017701+0000","last_active":"2024-03-04T16:19:30.365732+0000","last_peered":"2024-03-04T16:19:30.365732+0000","last_clean":"2024-03-04T16:19:30.365732+0000","last_became_active":"2024-03-04T16:19:04.771891+0000","last_became_peered":"2024-03-04T16:19:04.771891+0000","last_unstale":"2024-03-04T16:19:30.365732+0000","last_undegraded":"2024-03-04T16:19:30.365732+0000","last_fullsized":"2024-03-04T16:19:30.365732+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:18:15.488262+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.488262+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"351'9799","reported_seq":10427,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:34.044396+0000","last_change":"2024-03-04T16:19:05.532979+0000","last_active":"2024-03-04T16:19:34.044396+0000","last_peered":"2024-03-04T16:19:34.044396+0000","last_clean":"2024-03-04T16:18:20.331030+0000","last_became_active":"2024-03-04T16:19:04.881336+0000","last_became_peered":"2024-03-04T16:19:04.881336+0000","last_unstale":"2024-03-04T16:19:34.044396+0000","last_undegraded":"2024-03-04T16:19:34.044396+0000","last_fullsized":"2024-03-04T16:19:34.044396+0000","mapping_epoch":337,"log_start":"312'4556","ondisk_log_start":"312'4556","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5243,"log_dups_size":0,"ondisk_log_size":5243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162168832,"num_objects":2475,"num_object_clones":0,"num_object_copies":7425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7179,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6063,"num_write_kb":273184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":7798784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[3,5,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":2475}],"blocked_by":[],"up_primary":1,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":619,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390448+0000","last_change":"2024-03-04T16:17:48.459727+0000","last_active":"2024-03-04T16:19:30.390448+0000","last_peered":"2024-03-04T16:19:30.390448+0000","last_clean":"2024-03-04T16:19:30.390448+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:19:30.390448+0000","last_undegraded":"2024-03-04T16:19:30.390448+0000","last_fullsized":"2024-03-04T16:19:30.390448+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.459594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.459594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367073+0000","last_change":"2024-03-04T16:19:04.778249+0000","last_active":"2024-03-04T16:19:30.367073+0000","last_peered":"2024-03-04T16:19:30.367073+0000","last_clean":"2024-03-04T16:19:30.367073+0000","last_became_active":"2024-03-04T16:19:04.777261+0000","last_became_peered":"2024-03-04T16:19:04.777261+0000","last_unstale":"2024-03-04T16:19:30.367073+0000","last_undegraded":"2024-03-04T16:19:30.367073+0000","last_fullsized":"2024-03-04T16:19:30.367073+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.414689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.414689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":643,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367317+0000","last_change":"2024-03-04T16:19:04.759076+0000","last_active":"2024-03-04T16:19:30.367317+0000","last_peered":"2024-03-04T16:19:30.367317+0000","last_clean":"2024-03-04T16:19:30.367317+0000","last_became_active":"2024-03-04T16:19:04.758779+0000","last_became_peered":"2024-03-04T16:19:04.758779+0000","last_unstale":"2024-03-04T16:19:30.367317+0000","last_undegraded":"2024-03-04T16:19:30.367317+0000","last_fullsized":"2024-03-04T16:19:30.367317+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:17.538492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.538492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":659,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.384086+0000","last_change":"2024-03-04T16:17:03.702758+0000","last_active":"2024-03-04T16:19:30.384086+0000","last_peered":"2024-03-04T16:19:30.384086+0000","last_clean":"2024-03-04T16:19:30.384086+0000","last_became_active":"2024-03-04T16:02:45.750191+0000","last_became_peered":"2024-03-04T16:02:45.750191+0000","last_unstale":"2024-03-04T16:19:30.384086+0000","last_undegraded":"2024-03-04T16:19:30.384086+0000","last_fullsized":"2024-03-04T16:19:30.384086+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:17:03.702661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.702661+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"351'9667","reported_seq":16106,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:32.805607+0000","last_change":"2024-03-04T16:19:17.384903+0000","last_active":"2024-03-04T16:19:32.805607+0000","last_peered":"2024-03-04T16:19:32.805607+0000","last_clean":"2024-03-04T16:19:32.805607+0000","last_became_active":"2024-03-04T16:19:17.373869+0000","last_became_peered":"2024-03-04T16:19:17.373869+0000","last_unstale":"2024-03-04T16:19:32.805607+0000","last_undegraded":"2024-03-04T16:19:32.805607+0000","last_fullsized":"2024-03-04T16:19:32.805607+0000","mapping_epoch":345,"log_start":"312'4533","ondisk_log_start":"312'4533","created":320,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:37.807871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161284096,"num_objects":2461,"num_object_clones":0,"num_object_copies":7383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5941,"num_write_kb":268864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3144,"num_bytes_recovered":206045184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":649,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364174+0000","last_change":"2024-03-04T16:19:06.576246+0000","last_active":"2024-03-04T16:19:30.364174+0000","last_peered":"2024-03-04T16:19:30.364174+0000","last_clean":"2024-03-04T16:19:30.364174+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:19:30.364174+0000","last_undegraded":"2024-03-04T16:19:30.364174+0000","last_fullsized":"2024-03-04T16:19:30.364174+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:19:06.576128+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:19:06.576128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:14.941118+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"338'11","reported_seq":661,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370455+0000","last_change":"2024-03-04T16:19:04.839942+0000","last_active":"2024-03-04T16:19:30.370455+0000","last_peered":"2024-03-04T16:19:30.370455+0000","last_clean":"2024-03-04T16:19:30.370455+0000","last_became_active":"2024-03-04T16:19:04.777474+0000","last_became_peered":"2024-03-04T16:19:04.777474+0000","last_unstale":"2024-03-04T16:19:30.370455+0000","last_undegraded":"2024-03-04T16:19:30.370455+0000","last_fullsized":"2024-03-04T16:19:30.370455+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'10","last_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:17:08.205209+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":640,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.379516+0000","last_change":"2024-03-04T16:18:16.465800+0000","last_active":"2024-03-04T16:19:30.379516+0000","last_peered":"2024-03-04T16:19:30.379516+0000","last_clean":"2024-03-04T16:19:30.379516+0000","last_became_active":"2024-03-04T16:02:43.710677+0000","last_became_peered":"2024-03-04T16:02:43.710677+0000","last_unstale":"2024-03-04T16:19:30.379516+0000","last_undegraded":"2024-03-04T16:19:30.379516+0000","last_fullsized":"2024-03-04T16:19:30.379516+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:18:16.465669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.465669+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":503,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390130+0000","last_change":"2024-03-04T16:16:08.556672+0000","last_active":"2024-03-04T16:19:30.390130+0000","last_peered":"2024-03-04T16:19:30.390130+0000","last_clean":"2024-03-04T16:19:30.390130+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:19:30.390130+0000","last_undegraded":"2024-03-04T16:19:30.390130+0000","last_fullsized":"2024-03-04T16:19:30.390130+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:08.553910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.553910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"351'11412","reported_seq":19348,"reported_epoch":351,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-04T16:19:33.066490+0000","last_change":"2024-03-04T16:19:22.381323+0000","last_active":"2024-03-04T16:19:33.066490+0000","last_peered":"2024-03-04T16:19:33.066490+0000","last_clean":"2024-03-04T16:19:33.066490+0000","last_became_active":"2024-03-04T16:18:53.406638+0000","last_became_peered":"2024-03-04T16:18:53.406638+0000","last_unstale":"2024-03-04T16:19:33.066490+0000","last_undegraded":"2024-03-04T16:19:33.066490+0000","last_fullsized":"2024-03-04T16:19:33.066490+0000","mapping_epoch":331,"log_start":"316'6286","ondisk_log_start":"316'6286","created":320,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":1749,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 10s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158040064,"num_objects":2412,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5974,"num_write_kb":268320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3865,"num_bytes_recovered":253296640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":619,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.378296+0000","last_change":"2024-03-04T16:17:03.492110+0000","last_active":"2024-03-04T16:19:30.378296+0000","last_peered":"2024-03-04T16:19:30.378296+0000","last_clean":"2024-03-04T16:19:30.378296+0000","last_became_active":"2024-03-04T16:02:49.790413+0000","last_became_peered":"2024-03-04T16:02:49.790413+0000","last_unstale":"2024-03-04T16:19:30.378296+0000","last_undegraded":"2024-03-04T16:19:30.378296+0000","last_fullsized":"2024-03-04T16:19:30.378296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":152,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.389760+0000","last_change":"2024-03-04T16:19:04.848756+0000","last_active":"2024-03-04T16:19:30.389760+0000","last_peered":"2024-03-04T16:19:30.389760+0000","last_clean":"2024-03-04T16:19:30.389760+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:19:30.389760+0000","last_undegraded":"2024-03-04T16:19:30.389760+0000","last_fullsized":"2024-03-04T16:19:30.389760+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:00.852665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.852665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381050+0000","last_change":"2024-03-04T16:16:08.076532+0000","last_active":"2024-03-04T16:19:30.381050+0000","last_peered":"2024-03-04T16:19:30.381050+0000","last_clean":"2024-03-04T16:19:30.381050+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:19:30.381050+0000","last_undegraded":"2024-03-04T16:19:30.381050+0000","last_fullsized":"2024-03-04T16:19:30.381050+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.076402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":414,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369255+0000","last_change":"2024-03-04T16:17:06.624302+0000","last_active":"2024-03-04T16:19:30.369255+0000","last_peered":"2024-03-04T16:19:30.369255+0000","last_clean":"2024-03-04T16:19:30.369255+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:19:30.369255+0000","last_undegraded":"2024-03-04T16:19:30.369255+0000","last_fullsized":"2024-03-04T16:19:30.369255+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:06.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.624154+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":648,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363898+0000","last_change":"2024-03-04T16:19:05.624025+0000","last_active":"2024-03-04T16:19:30.363898+0000","last_peered":"2024-03-04T16:19:30.363898+0000","last_clean":"2024-03-04T16:19:30.363898+0000","last_became_active":"2024-03-04T16:19:04.749519+0000","last_became_peered":"2024-03-04T16:19:04.749519+0000","last_unstale":"2024-03-04T16:19:30.363898+0000","last_undegraded":"2024-03-04T16:19:30.363898+0000","last_fullsized":"2024-03-04T16:19:30.363898+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:05.623916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:19:05.623916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:12.725060+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":343,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366200+0000","last_change":"2024-03-04T16:15:58.573709+0000","last_active":"2024-03-04T16:19:30.366200+0000","last_peered":"2024-03-04T16:19:30.366200+0000","last_clean":"2024-03-04T16:19:30.366200+0000","last_became_active":"2024-03-04T16:10:29.237754+0000","last_became_peered":"2024-03-04T16:10:29.237754+0000","last_unstale":"2024-03-04T16:19:30.366200+0000","last_undegraded":"2024-03-04T16:19:30.366200+0000","last_fullsized":"2024-03-04T16:19:30.366200+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.573599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.573599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":475,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390652+0000","last_change":"2024-03-04T16:17:58.626787+0000","last_active":"2024-03-04T16:19:30.390652+0000","last_peered":"2024-03-04T16:19:30.390652+0000","last_clean":"2024-03-04T16:19:30.390652+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:19:30.390652+0000","last_undegraded":"2024-03-04T16:19:30.390652+0000","last_fullsized":"2024-03-04T16:19:30.390652+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:58.626660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.626660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":644,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381149+0000","last_change":"2024-03-04T16:16:59.675333+0000","last_active":"2024-03-04T16:19:30.381149+0000","last_peered":"2024-03-04T16:19:30.381149+0000","last_clean":"2024-03-04T16:19:30.381149+0000","last_became_active":"2024-03-04T16:10:29.251290+0000","last_became_peered":"2024-03-04T16:10:29.251290+0000","last_unstale":"2024-03-04T16:19:30.381149+0000","last_undegraded":"2024-03-04T16:19:30.381149+0000","last_fullsized":"2024-03-04T16:19:30.381149+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:16:59.675176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.675176+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":636,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.385475+0000","last_change":"2024-03-04T16:19:04.781577+0000","last_active":"2024-03-04T16:19:30.385475+0000","last_peered":"2024-03-04T16:19:30.385475+0000","last_clean":"2024-03-04T16:19:30.385475+0000","last_became_active":"2024-03-04T16:19:04.781224+0000","last_became_peered":"2024-03-04T16:19:04.781224+0000","last_unstale":"2024-03-04T16:19:30.385475+0000","last_undegraded":"2024-03-04T16:19:30.385475+0000","last_fullsized":"2024-03-04T16:19:30.385475+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:14.574861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.574861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:31.558959+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":638,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364310+0000","last_change":"2024-03-04T16:19:08.601618+0000","last_active":"2024-03-04T16:19:30.364310+0000","last_peered":"2024-03-04T16:19:30.364310+0000","last_clean":"2024-03-04T16:19:30.364310+0000","last_became_active":"2024-03-04T16:10:29.247501+0000","last_became_peered":"2024-03-04T16:10:29.247501+0000","last_unstale":"2024-03-04T16:19:30.364310+0000","last_undegraded":"2024-03-04T16:19:30.364310+0000","last_fullsized":"2024-03-04T16:19:30.364310+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:08.601393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:19:08.601393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:12.836461+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":638,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369886+0000","last_change":"2024-03-04T16:17:01.618230+0000","last_active":"2024-03-04T16:19:30.369886+0000","last_peered":"2024-03-04T16:19:30.369886+0000","last_clean":"2024-03-04T16:19:30.369886+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:19:30.369886+0000","last_undegraded":"2024-03-04T16:19:30.369886+0000","last_fullsized":"2024-03-04T16:19:30.369886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.618146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.618146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":669,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381874+0000","last_change":"2024-03-04T16:18:06.391143+0000","last_active":"2024-03-04T16:19:30.381874+0000","last_peered":"2024-03-04T16:19:30.381874+0000","last_clean":"2024-03-04T16:19:30.381874+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:19:30.381874+0000","last_undegraded":"2024-03-04T16:19:30.381874+0000","last_fullsized":"2024-03-04T16:19:30.381874+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:18:06.391045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:06.391045+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":476,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381391+0000","last_change":"2024-03-04T16:18:10.894569+0000","last_active":"2024-03-04T16:19:30.381391+0000","last_peered":"2024-03-04T16:19:30.381391+0000","last_clean":"2024-03-04T16:19:30.381391+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:19:30.381391+0000","last_undegraded":"2024-03-04T16:19:30.381391+0000","last_fullsized":"2024-03-04T16:19:30.381391+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:10.893919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:18:10.893919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":626,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.385253+0000","last_change":"2024-03-04T16:16:05.072152+0000","last_active":"2024-03-04T16:19:30.385253+0000","last_peered":"2024-03-04T16:19:30.385253+0000","last_clean":"2024-03-04T16:19:30.385253+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:19:30.385253+0000","last_undegraded":"2024-03-04T16:19:30.385253+0000","last_fullsized":"2024-03-04T16:19:30.385253+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:05.071956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:05.071956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364498+0000","last_change":"2024-03-04T16:17:56.238575+0000","last_active":"2024-03-04T16:19:30.364498+0000","last_peered":"2024-03-04T16:19:30.364498+0000","last_clean":"2024-03-04T16:19:30.364498+0000","last_became_active":"2024-03-04T16:02:43.711223+0000","last_became_peered":"2024-03-04T16:02:43.711223+0000","last_unstale":"2024-03-04T16:19:30.364498+0000","last_undegraded":"2024-03-04T16:19:30.364498+0000","last_fullsized":"2024-03-04T16:19:30.364498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:56.238364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:56.238364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":713,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377552+0000","last_change":"2024-03-04T16:19:04.909849+0000","last_active":"2024-03-04T16:19:30.377552+0000","last_peered":"2024-03-04T16:19:30.377552+0000","last_clean":"2024-03-04T16:19:30.377552+0000","last_became_active":"2024-03-04T16:19:04.791277+0000","last_became_peered":"2024-03-04T16:19:04.791277+0000","last_unstale":"2024-03-04T16:19:30.377552+0000","last_undegraded":"2024-03-04T16:19:30.377552+0000","last_fullsized":"2024-03-04T16:19:30.377552+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:51.289785+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:17:51.289785+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":624,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.380711+0000","last_change":"2024-03-04T16:18:17.376026+0000","last_active":"2024-03-04T16:19:30.380711+0000","last_peered":"2024-03-04T16:19:30.380711+0000","last_clean":"2024-03-04T16:19:30.380711+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:19:30.380711+0000","last_undegraded":"2024-03-04T16:19:30.380711+0000","last_fullsized":"2024-03-04T16:19:30.380711+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:17.375837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:18:17.375837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:34.882785+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377377+0000","last_change":"2024-03-04T16:17:54.262357+0000","last_active":"2024-03-04T16:19:30.377377+0000","last_peered":"2024-03-04T16:19:30.377377+0000","last_clean":"2024-03-04T16:19:30.377377+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:19:30.377377+0000","last_undegraded":"2024-03-04T16:19:30.377377+0000","last_fullsized":"2024-03-04T16:19:30.377377+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:54.262228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.262228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":644,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366788+0000","last_change":"2024-03-04T16:19:04.778327+0000","last_active":"2024-03-04T16:19:30.366788+0000","last_peered":"2024-03-04T16:19:30.366788+0000","last_clean":"2024-03-04T16:19:30.366788+0000","last_became_active":"2024-03-04T16:19:04.777392+0000","last_became_peered":"2024-03-04T16:19:04.777392+0000","last_unstale":"2024-03-04T16:19:30.366788+0000","last_undegraded":"2024-03-04T16:19:30.366788+0000","last_fullsized":"2024-03-04T16:19:30.366788+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.350240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.350240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"316'24","reported_seq":690,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367069+0000","last_change":"2024-03-04T16:18:04.512732+0000","last_active":"2024-03-04T16:19:30.367069+0000","last_peered":"2024-03-04T16:19:30.367069+0000","last_clean":"2024-03-04T16:19:30.367069+0000","last_became_active":"2024-03-04T16:10:29.326847+0000","last_became_peered":"2024-03-04T16:10:29.326847+0000","last_unstale":"2024-03-04T16:19:30.367069+0000","last_undegraded":"2024-03-04T16:19:30.367069+0000","last_fullsized":"2024-03-04T16:19:30.367069+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'24","last_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_deep_scrub":"316'24","last_deep_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_clean_scrub_stamp":"2024-03-04T16:18:04.512502+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":509,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364154+0000","last_change":"2024-03-04T16:19:22.631070+0000","last_active":"2024-03-04T16:19:30.364154+0000","last_peered":"2024-03-04T16:19:30.364154+0000","last_clean":"2024-03-04T16:19:30.364154+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:19:30.364154+0000","last_undegraded":"2024-03-04T16:19:30.364154+0000","last_fullsized":"2024-03-04T16:19:30.364154+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:22.630934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:19:22.630934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:51.708453+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":854,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370544+0000","last_change":"2024-03-04T16:16:59.613637+0000","last_active":"2024-03-04T16:19:30.370544+0000","last_peered":"2024-03-04T16:19:30.370544+0000","last_clean":"2024-03-04T16:19:30.370544+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:19:30.370544+0000","last_undegraded":"2024-03-04T16:19:30.370544+0000","last_fullsized":"2024-03-04T16:19:30.370544+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:16:59.613549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.613549+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":684,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363739+0000","last_change":"2024-03-04T16:19:09.596192+0000","last_active":"2024-03-04T16:19:30.363739+0000","last_peered":"2024-03-04T16:19:30.363739+0000","last_clean":"2024-03-04T16:19:30.363739+0000","last_became_active":"2024-03-04T16:19:04.749611+0000","last_became_peered":"2024-03-04T16:19:04.749611+0000","last_unstale":"2024-03-04T16:19:30.363739+0000","last_undegraded":"2024-03-04T16:19:30.363739+0000","last_fullsized":"2024-03-04T16:19:30.363739+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:19:09.596049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:19:09.596049+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:30.841075+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":678,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.383351+0000","last_change":"2024-03-04T16:16:53.775738+0000","last_active":"2024-03-04T16:19:30.383351+0000","last_peered":"2024-03-04T16:19:30.383351+0000","last_clean":"2024-03-04T16:19:30.383351+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:19:30.383351+0000","last_undegraded":"2024-03-04T16:19:30.383351+0000","last_fullsized":"2024-03-04T16:19:30.383351+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:53.775609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.775609+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370520+0000","last_change":"2024-03-04T16:19:04.777973+0000","last_active":"2024-03-04T16:19:30.370520+0000","last_peered":"2024-03-04T16:19:30.370520+0000","last_clean":"2024-03-04T16:19:30.370520+0000","last_became_active":"2024-03-04T16:19:04.777178+0000","last_became_peered":"2024-03-04T16:19:04.777178+0000","last_unstale":"2024-03-04T16:19:30.370520+0000","last_undegraded":"2024-03-04T16:19:30.370520+0000","last_fullsized":"2024-03-04T16:19:30.370520+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.474636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.474636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"338'11","reported_seq":848,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.383685+0000","last_change":"2024-03-04T16:19:04.890313+0000","last_active":"2024-03-04T16:19:30.383685+0000","last_peered":"2024-03-04T16:19:30.383685+0000","last_clean":"2024-03-04T16:19:30.383685+0000","last_became_active":"2024-03-04T16:19:04.780989+0000","last_became_peered":"2024-03-04T16:19:04.780989+0000","last_unstale":"2024-03-04T16:19:30.383685+0000","last_undegraded":"2024-03-04T16:19:30.383685+0000","last_fullsized":"2024-03-04T16:19:30.383685+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.365318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":533,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363641+0000","last_change":"2024-03-04T16:17:47.360887+0000","last_active":"2024-03-04T16:19:30.363641+0000","last_peered":"2024-03-04T16:19:30.363641+0000","last_clean":"2024-03-04T16:19:30.363641+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:19:30.363641+0000","last_undegraded":"2024-03-04T16:19:30.363641+0000","last_fullsized":"2024-03-04T16:19:30.363641+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:17:47.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.360747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":659,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370561+0000","last_change":"2024-03-04T16:18:01.423163+0000","last_active":"2024-03-04T16:19:30.370561+0000","last_peered":"2024-03-04T16:19:30.370561+0000","last_clean":"2024-03-04T16:19:30.370561+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:19:30.370561+0000","last_undegraded":"2024-03-04T16:19:30.370561+0000","last_fullsized":"2024-03-04T16:19:30.370561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:18:01.423072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:01.423072+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"351'10048","reported_seq":20890,"reported_epoch":351,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:19:33.830552+0000","last_change":"2024-03-04T16:19:05.019254+0000","last_active":"2024-03-04T16:19:33.830552+0000","last_peered":"2024-03-04T16:19:33.830552+0000","last_clean":"2024-03-04T16:18:20.343788+0000","last_became_active":"2024-03-04T16:19:04.904088+0000","last_became_peered":"2024-03-04T16:19:04.904088+0000","last_unstale":"2024-03-04T16:19:33.830552+0000","last_undegraded":"2024-03-04T16:19:33.830552+0000","last_fullsized":"2024-03-04T16:19:33.830552+0000","mapping_epoch":337,"log_start":"309'3867","ondisk_log_start":"309'3867","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6181,"log_dups_size":0,"ondisk_log_size":6181,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161579008,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2774,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6226,"num_write_kb":278112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":233701376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.365888+0000","last_change":"2024-03-04T16:19:04.750266+0000","last_active":"2024-03-04T16:19:30.365888+0000","last_peered":"2024-03-04T16:19:30.365888+0000","last_clean":"2024-03-04T16:19:30.365888+0000","last_became_active":"2024-03-04T16:19:04.749973+0000","last_became_peered":"2024-03-04T16:19:04.749973+0000","last_unstale":"2024-03-04T16:19:30.365888+0000","last_undegraded":"2024-03-04T16:19:30.365888+0000","last_fullsized":"2024-03-04T16:19:30.365888+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":615,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390979+0000","last_change":"2024-03-04T16:17:55.208057+0000","last_active":"2024-03-04T16:19:30.390979+0000","last_peered":"2024-03-04T16:19:30.390979+0000","last_clean":"2024-03-04T16:19:30.390979+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:19:30.390979+0000","last_undegraded":"2024-03-04T16:19:30.390979+0000","last_fullsized":"2024-03-04T16:19:30.390979+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:55.207977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.207977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.702611+0000","last_change":"2024-03-04T16:19:30.702646+0000","last_active":"2024-03-04T16:19:30.702611+0000","last_peered":"2024-03-04T16:19:30.702611+0000","last_clean":"2024-03-04T16:19:30.702611+0000","last_became_active":"2024-03-04T16:10:29.241948+0000","last_became_peered":"2024-03-04T16:10:29.241948+0000","last_unstale":"2024-03-04T16:19:30.702611+0000","last_undegraded":"2024-03-04T16:19:30.702611+0000","last_fullsized":"2024-03-04T16:19:30.702611+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_clean_scrub_stamp":"2024-03-04T16:19:30.702539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:41.833285+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":494,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390227+0000","last_change":"2024-03-04T16:16:53.856464+0000","last_active":"2024-03-04T16:19:30.390227+0000","last_peered":"2024-03-04T16:19:30.390227+0000","last_clean":"2024-03-04T16:19:30.390227+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:19:30.390227+0000","last_undegraded":"2024-03-04T16:19:30.390227+0000","last_fullsized":"2024-03-04T16:19:30.390227+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:16:53.856361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.856361+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.19","version":"351'9899","reported_seq":12019,"reported_epoch":351,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:19:33.077209+0000","last_change":"2024-03-04T16:19:22.575184+0000","last_active":"2024-03-04T16:19:33.077209+0000","last_peered":"2024-03-04T16:19:33.077209+0000","last_clean":"2024-03-04T16:18:20.360662+0000","last_became_active":"2024-03-04T16:18:22.469940+0000","last_became_peered":"2024-03-04T16:18:22.469940+0000","last_unstale":"2024-03-04T16:19:33.077209+0000","last_undegraded":"2024-03-04T16:19:33.077209+0000","last_fullsized":"2024-03-04T16:19:33.077209+0000","mapping_epoch":323,"log_start":"308'3205","ondisk_log_start":"308'3205","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6694,"log_dups_size":0,"ondisk_log_size":6694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165216256,"num_objects":2521,"num_object_clones":0,"num_object_copies":7563,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3818,"num_objects_unfound":0,"num_objects_dirty":2521,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6127,"num_write_kb":276736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":43188224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[5,4,7],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5,7","objects":2521}],"blocked_by":[],"up_primary":1,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":633,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364423+0000","last_change":"2024-03-04T16:16:56.475032+0000","last_active":"2024-03-04T16:19:30.364423+0000","last_peered":"2024-03-04T16:19:30.364423+0000","last_clean":"2024-03-04T16:19:30.364423+0000","last_became_active":"2024-03-04T16:10:29.369350+0000","last_became_peered":"2024-03-04T16:10:29.369350+0000","last_unstale":"2024-03-04T16:19:30.364423+0000","last_undegraded":"2024-03-04T16:19:30.364423+0000","last_fullsized":"2024-03-04T16:19:30.364423+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:56.474881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:16:56.474881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":622,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369588+0000","last_change":"2024-03-04T16:15:56.777453+0000","last_active":"2024-03-04T16:19:30.369588+0000","last_peered":"2024-03-04T16:19:30.369588+0000","last_clean":"2024-03-04T16:19:30.369588+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:19:30.369588+0000","last_undegraded":"2024-03-04T16:19:30.369588+0000","last_fullsized":"2024-03-04T16:19:30.369588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:56.777347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.777347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":537,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.388400+0000","last_change":"2024-03-04T16:16:51.831340+0000","last_active":"2024-03-04T16:19:30.388400+0000","last_peered":"2024-03-04T16:19:30.388400+0000","last_clean":"2024-03-04T16:19:30.388400+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:19:30.388400+0000","last_undegraded":"2024-03-04T16:19:30.388400+0000","last_fullsized":"2024-03-04T16:19:30.388400+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:51.830496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:16:51.830496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":640,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390547+0000","last_change":"2024-03-04T16:18:01.357560+0000","last_active":"2024-03-04T16:19:30.390547+0000","last_peered":"2024-03-04T16:19:30.390547+0000","last_clean":"2024-03-04T16:19:30.390547+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:19:30.390547+0000","last_undegraded":"2024-03-04T16:19:30.390547+0000","last_fullsized":"2024-03-04T16:19:30.390547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:18:01.357413+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:18:01.357413+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"351'9814","reported_seq":9984,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.023838+0000","last_change":"2024-03-04T16:19:21.986552+0000","last_active":"2024-03-04T16:19:33.023838+0000","last_peered":"2024-03-04T16:19:33.023838+0000","last_clean":"2024-03-04T16:19:33.023838+0000","last_became_active":"2024-03-04T16:18:19.393633+0000","last_became_peered":"2024-03-04T16:18:19.393633+0000","last_unstale":"2024-03-04T16:19:33.023838+0000","last_undegraded":"2024-03-04T16:19:33.023838+0000","last_fullsized":"2024-03-04T16:19:33.023838+0000","mapping_epoch":296,"log_start":"308'3102","ondisk_log_start":"308'3102","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'9547","last_scrub_stamp":"2024-03-04T16:19:21.986416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:19:21.986416+0000","objects_scrubbed":2778,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:41.431638+0000","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":270,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368712+0000","last_change":"2024-03-04T16:19:04.784783+0000","last_active":"2024-03-04T16:19:30.368712+0000","last_peered":"2024-03-04T16:19:30.368712+0000","last_clean":"2024-03-04T16:19:30.368712+0000","last_became_active":"2024-03-04T16:19:04.784559+0000","last_became_peered":"2024-03-04T16:19:04.784559+0000","last_unstale":"2024-03-04T16:19:30.368712+0000","last_undegraded":"2024-03-04T16:19:30.368712+0000","last_fullsized":"2024-03-04T16:19:30.368712+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.446216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.446216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:42.863969+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":676,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364649+0000","last_change":"2024-03-04T16:17:07.251362+0000","last_active":"2024-03-04T16:19:30.364649+0000","last_peered":"2024-03-04T16:19:30.364649+0000","last_clean":"2024-03-04T16:19:30.364649+0000","last_became_active":"2024-03-04T16:07:36.207761+0000","last_became_peered":"2024-03-04T16:07:36.207761+0000","last_unstale":"2024-03-04T16:19:30.364649+0000","last_undegraded":"2024-03-04T16:19:30.364649+0000","last_fullsized":"2024-03-04T16:19:30.364649+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:17:07.251155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.251155+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":700,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367953+0000","last_change":"2024-03-04T16:17:54.552434+0000","last_active":"2024-03-04T16:19:30.367953+0000","last_peered":"2024-03-04T16:19:30.367953+0000","last_clean":"2024-03-04T16:19:30.367953+0000","last_became_active":"2024-03-04T16:10:29.330859+0000","last_became_peered":"2024-03-04T16:10:29.330859+0000","last_unstale":"2024-03-04T16:19:30.367953+0000","last_undegraded":"2024-03-04T16:19:30.367953+0000","last_fullsized":"2024-03-04T16:19:30.367953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:17:54.552272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.552272+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"195'9","reported_seq":852,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.365008+0000","last_change":"2024-03-04T16:17:09.268256+0000","last_active":"2024-03-04T16:19:30.365008+0000","last_peered":"2024-03-04T16:19:30.365008+0000","last_clean":"2024-03-04T16:19:30.365008+0000","last_became_active":"2024-03-04T16:10:29.242717+0000","last_became_peered":"2024-03-04T16:10:29.242717+0000","last_unstale":"2024-03-04T16:19:30.365008+0000","last_undegraded":"2024-03-04T16:19:30.365008+0000","last_fullsized":"2024-03-04T16:19:30.365008+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:17:09.268137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:09.268137+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"351'9778","reported_seq":14009,"reported_epoch":351,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:19:34.048369+0000","last_change":"2024-03-04T16:19:16.379597+0000","last_active":"2024-03-04T16:19:34.048369+0000","last_peered":"2024-03-04T16:19:34.048369+0000","last_clean":"2024-03-04T16:19:03.683472+0000","last_became_active":"2024-03-04T16:19:05.502124+0000","last_became_peered":"2024-03-04T16:19:05.502124+0000","last_unstale":"2024-03-04T16:19:34.048369+0000","last_undegraded":"2024-03-04T16:19:34.048369+0000","last_fullsized":"2024-03-04T16:19:34.048369+0000","mapping_epoch":338,"log_start":"312'4478","ondisk_log_start":"312'4478","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5300,"log_dups_size":0,"ondisk_log_size":5300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159252480,"num_objects":2430,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":565,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6062,"num_write_kb":271744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2049,"num_bytes_recovered":134283264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":2430}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":612,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.385499+0000","last_change":"2024-03-04T16:16:18.274771+0000","last_active":"2024-03-04T16:19:30.385499+0000","last_peered":"2024-03-04T16:19:30.385499+0000","last_clean":"2024-03-04T16:19:30.385499+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:19:30.385499+0000","last_undegraded":"2024-03-04T16:19:30.385499+0000","last_fullsized":"2024-03-04T16:19:30.385499+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.274659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.274659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":648,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364384+0000","last_change":"2024-03-04T16:19:15.612734+0000","last_active":"2024-03-04T16:19:30.364384+0000","last_peered":"2024-03-04T16:19:30.364384+0000","last_clean":"2024-03-04T16:19:30.364384+0000","last_became_active":"2024-03-04T16:02:43.710498+0000","last_became_peered":"2024-03-04T16:02:43.710498+0000","last_unstale":"2024-03-04T16:19:30.364384+0000","last_undegraded":"2024-03-04T16:19:30.364384+0000","last_fullsized":"2024-03-04T16:19:30.364384+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:15.612623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:19:15.612623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:20.485280+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":660,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366632+0000","last_change":"2024-03-04T16:16:08.552789+0000","last_active":"2024-03-04T16:19:30.366632+0000","last_peered":"2024-03-04T16:19:30.366632+0000","last_clean":"2024-03-04T16:19:30.366632+0000","last_became_active":"2024-03-04T16:02:45.745485+0000","last_became_peered":"2024-03-04T16:02:45.745485+0000","last_unstale":"2024-03-04T16:19:30.366632+0000","last_undegraded":"2024-03-04T16:19:30.366632+0000","last_fullsized":"2024-03-04T16:19:30.366632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:16:08.552635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.552635+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367088+0000","last_change":"2024-03-04T16:19:04.768572+0000","last_active":"2024-03-04T16:19:30.367088+0000","last_peered":"2024-03-04T16:19:30.367088+0000","last_clean":"2024-03-04T16:19:30.367088+0000","last_became_active":"2024-03-04T16:19:04.768239+0000","last_became_peered":"2024-03-04T16:19:04.768239+0000","last_unstale":"2024-03-04T16:19:30.367088+0000","last_undegraded":"2024-03-04T16:19:30.367088+0000","last_fullsized":"2024-03-04T16:19:30.367088+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:16.337706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.337706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"351'9746","reported_seq":9876,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.822755+0000","last_change":"2024-03-04T16:18:19.375591+0000","last_active":"2024-03-04T16:19:33.822755+0000","last_peered":"2024-03-04T16:19:33.822755+0000","last_clean":"2024-03-04T16:19:33.822755+0000","last_became_active":"2024-03-04T16:18:19.373372+0000","last_became_peered":"2024-03-04T16:18:19.373372+0000","last_unstale":"2024-03-04T16:19:33.822755+0000","last_undegraded":"2024-03-04T16:19:33.822755+0000","last_fullsized":"2024-03-04T16:19:33.822755+0000","mapping_epoch":296,"log_start":"309'3500","ondisk_log_start":"309'3500","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6246,"log_dups_size":0,"ondisk_log_size":6246,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:36.011412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165478400,"num_objects":2525,"num_object_clones":0,"num_object_copies":7575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6007,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363449+0000","last_change":"2024-03-04T16:19:04.749645+0000","last_active":"2024-03-04T16:19:30.363449+0000","last_peered":"2024-03-04T16:19:30.363449+0000","last_clean":"2024-03-04T16:19:30.363449+0000","last_became_active":"2024-03-04T16:19:04.749223+0000","last_became_peered":"2024-03-04T16:19:04.749223+0000","last_unstale":"2024-03-04T16:19:30.363449+0000","last_undegraded":"2024-03-04T16:19:30.363449+0000","last_fullsized":"2024-03-04T16:19:30.363449+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:57.307587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:57.307587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368912+0000","last_change":"2024-03-04T16:19:04.880283+0000","last_active":"2024-03-04T16:19:30.368912+0000","last_peered":"2024-03-04T16:19:30.368912+0000","last_clean":"2024-03-04T16:19:30.368912+0000","last_became_active":"2024-03-04T16:19:04.780445+0000","last_became_peered":"2024-03-04T16:19:04.780445+0000","last_unstale":"2024-03-04T16:19:30.368912+0000","last_undegraded":"2024-03-04T16:19:30.368912+0000","last_fullsized":"2024-03-04T16:19:30.368912+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:15.744737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.744737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":646,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364595+0000","last_change":"2024-03-04T16:17:46.310631+0000","last_active":"2024-03-04T16:19:30.364595+0000","last_peered":"2024-03-04T16:19:30.364595+0000","last_clean":"2024-03-04T16:19:30.364595+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:19:30.364595+0000","last_undegraded":"2024-03-04T16:19:30.364595+0000","last_fullsized":"2024-03-04T16:19:30.364595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.310358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.310358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":447,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377207+0000","last_change":"2024-03-04T16:17:58.315576+0000","last_active":"2024-03-04T16:19:30.377207+0000","last_peered":"2024-03-04T16:19:30.377207+0000","last_clean":"2024-03-04T16:19:30.377207+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:19:30.377207+0000","last_undegraded":"2024-03-04T16:19:30.377207+0000","last_fullsized":"2024-03-04T16:19:30.377207+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:17:58.315405+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.315405+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"351'9825","reported_seq":12058,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:33.770371+0000","last_change":"2024-03-04T16:19:16.378895+0000","last_active":"2024-03-04T16:19:33.770371+0000","last_peered":"2024-03-04T16:19:33.770371+0000","last_clean":"2024-03-04T16:19:03.722374+0000","last_became_active":"2024-03-04T16:19:05.450089+0000","last_became_peered":"2024-03-04T16:19:05.450089+0000","last_unstale":"2024-03-04T16:19:33.770371+0000","last_undegraded":"2024-03-04T16:19:33.770371+0000","last_fullsized":"2024-03-04T16:19:33.770371+0000","mapping_epoch":338,"log_start":"309'3600","ondisk_log_start":"309'3600","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6225,"log_dups_size":0,"ondisk_log_size":6225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166363136,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1682,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":274656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1048,"num_bytes_recovered":68681728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[0,3,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":2538}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":521,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363558+0000","last_change":"2024-03-04T16:19:04.779974+0000","last_active":"2024-03-04T16:19:30.363558+0000","last_peered":"2024-03-04T16:19:30.363558+0000","last_clean":"2024-03-04T16:19:30.363558+0000","last_became_active":"2024-03-04T16:19:04.779570+0000","last_became_peered":"2024-03-04T16:19:04.779570+0000","last_unstale":"2024-03-04T16:19:30.363558+0000","last_undegraded":"2024-03-04T16:19:30.363558+0000","last_fullsized":"2024-03-04T16:19:30.363558+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:16.573055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.573055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366585+0000","last_change":"2024-03-04T16:19:04.769294+0000","last_active":"2024-03-04T16:19:30.366585+0000","last_peered":"2024-03-04T16:19:30.366585+0000","last_clean":"2024-03-04T16:19:30.366585+0000","last_became_active":"2024-03-04T16:19:04.768973+0000","last_became_peered":"2024-03-04T16:19:04.768973+0000","last_unstale":"2024-03-04T16:19:30.366585+0000","last_undegraded":"2024-03-04T16:19:30.366585+0000","last_fullsized":"2024-03-04T16:19:30.366585+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:52.337129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:17:52.337129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":645,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368919+0000","last_change":"2024-03-04T16:18:11.409686+0000","last_active":"2024-03-04T16:19:30.368919+0000","last_peered":"2024-03-04T16:19:30.368919+0000","last_clean":"2024-03-04T16:19:30.368919+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:19:30.368919+0000","last_undegraded":"2024-03-04T16:19:30.368919+0000","last_fullsized":"2024-03-04T16:19:30.368919+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:11.409565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:18:11.409565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:34.365273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":495,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.380778+0000","last_change":"2024-03-04T16:16:49.762334+0000","last_active":"2024-03-04T16:19:30.380778+0000","last_peered":"2024-03-04T16:19:30.380778+0000","last_clean":"2024-03-04T16:19:30.380778+0000","last_became_active":"2024-03-04T16:07:36.226330+0000","last_became_peered":"2024-03-04T16:07:36.226330+0000","last_unstale":"2024-03-04T16:19:30.380778+0000","last_undegraded":"2024-03-04T16:19:30.380778+0000","last_fullsized":"2024-03-04T16:19:30.380778+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:16:49.762239+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.762239+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"351'11950","reported_seq":12083,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:33.813094+0000","last_change":"2024-03-04T16:19:05.551460+0000","last_active":"2024-03-04T16:19:33.813094+0000","last_peered":"2024-03-04T16:19:33.813094+0000","last_clean":"2024-03-04T16:19:03.720570+0000","last_became_active":"2024-03-04T16:19:05.473830+0000","last_became_peered":"2024-03-04T16:19:05.473830+0000","last_unstale":"2024-03-04T16:19:33.813094+0000","last_undegraded":"2024-03-04T16:19:33.813094+0000","last_fullsized":"2024-03-04T16:19:33.813094+0000","mapping_epoch":338,"log_start":"324'7595","ondisk_log_start":"324'7595","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4886,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11950,"num_write_kb":538752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":4886}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":612,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.382842+0000","last_change":"2024-03-04T16:18:05.382514+0000","last_active":"2024-03-04T16:19:30.382842+0000","last_peered":"2024-03-04T16:19:30.382842+0000","last_clean":"2024-03-04T16:19:30.382842+0000","last_became_active":"2024-03-04T16:02:49.793185+0000","last_became_peered":"2024-03-04T16:02:49.793185+0000","last_unstale":"2024-03-04T16:19:30.382842+0000","last_undegraded":"2024-03-04T16:19:30.382842+0000","last_fullsized":"2024-03-04T16:19:30.382842+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:05.382402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:05.382402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:32.728998+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":653,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368815+0000","last_change":"2024-03-04T16:15:51.894739+0000","last_active":"2024-03-04T16:19:30.368815+0000","last_peered":"2024-03-04T16:19:30.368815+0000","last_clean":"2024-03-04T16:19:30.368815+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:19:30.368815+0000","last_undegraded":"2024-03-04T16:19:30.368815+0000","last_fullsized":"2024-03-04T16:19:30.368815+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:15:51.894452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.894452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":659,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366909+0000","last_change":"2024-03-04T16:18:08.651904+0000","last_active":"2024-03-04T16:19:30.366909+0000","last_peered":"2024-03-04T16:19:30.366909+0000","last_clean":"2024-03-04T16:19:30.366909+0000","last_became_active":"2024-03-04T16:10:29.321391+0000","last_became_peered":"2024-03-04T16:10:29.321391+0000","last_unstale":"2024-03-04T16:19:30.366909+0000","last_undegraded":"2024-03-04T16:19:30.366909+0000","last_fullsized":"2024-03-04T16:19:30.366909+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:18:08.651791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"338'43","reported_seq":989,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377583+0000","last_change":"2024-03-04T16:19:04.909779+0000","last_active":"2024-03-04T16:19:30.377583+0000","last_peered":"2024-03-04T16:19:30.377583+0000","last_clean":"2024-03-04T16:19:30.377583+0000","last_became_active":"2024-03-04T16:19:04.769086+0000","last_became_peered":"2024-03-04T16:19:04.769086+0000","last_unstale":"2024-03-04T16:19:30.377583+0000","last_undegraded":"2024-03-04T16:19:30.377583+0000","last_fullsized":"2024-03-04T16:19:30.377583+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:18:14.457588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.457588+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":476,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.396727+0000","last_change":"2024-03-04T16:17:01.832332+0000","last_active":"2024-03-04T16:19:30.396727+0000","last_peered":"2024-03-04T16:19:30.396727+0000","last_clean":"2024-03-04T16:19:30.396727+0000","last_became_active":"2024-03-04T16:07:36.231061+0000","last_became_peered":"2024-03-04T16:07:36.231061+0000","last_unstale":"2024-03-04T16:19:30.396727+0000","last_undegraded":"2024-03-04T16:19:30.396727+0000","last_fullsized":"2024-03-04T16:19:30.396727+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.832115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.832115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"351'9863","reported_seq":15979,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.055818+0000","last_change":"2024-03-04T16:19:22.405338+0000","last_active":"2024-03-04T16:19:33.055818+0000","last_peered":"2024-03-04T16:19:33.055818+0000","last_clean":"2024-03-04T16:19:33.055818+0000","last_became_active":"2024-03-04T16:19:22.395019+0000","last_became_peered":"2024-03-04T16:19:22.395019+0000","last_unstale":"2024-03-04T16:19:33.055818+0000","last_undegraded":"2024-03-04T16:19:33.055818+0000","last_fullsized":"2024-03-04T16:19:33.055818+0000","mapping_epoch":348,"log_start":"308'3064","ondisk_log_start":"308'3064","created":296,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6799,"log_dups_size":0,"ondisk_log_size":6799,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:33.465682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164691968,"num_objects":2513,"num_object_clones":0,"num_object_copies":7539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6091,"num_write_kb":275328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2987,"num_bytes_recovered":195756032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":641,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364739+0000","last_change":"2024-03-04T16:19:04.756842+0000","last_active":"2024-03-04T16:19:30.364739+0000","last_peered":"2024-03-04T16:19:30.364739+0000","last_clean":"2024-03-04T16:19:30.364739+0000","last_became_active":"2024-03-04T16:19:04.756529+0000","last_became_peered":"2024-03-04T16:19:04.756529+0000","last_unstale":"2024-03-04T16:19:30.364739+0000","last_undegraded":"2024-03-04T16:19:30.364739+0000","last_fullsized":"2024-03-04T16:19:30.364739+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.390204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.390204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":659,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.386077+0000","last_change":"2024-03-04T16:19:04.796532+0000","last_active":"2024-03-04T16:19:30.386077+0000","last_peered":"2024-03-04T16:19:30.386077+0000","last_clean":"2024-03-04T16:19:30.386077+0000","last_became_active":"2024-03-04T16:19:04.796249+0000","last_became_peered":"2024-03-04T16:19:04.796249+0000","last_unstale":"2024-03-04T16:19:30.386077+0000","last_undegraded":"2024-03-04T16:19:30.386077+0000","last_fullsized":"2024-03-04T16:19:30.386077+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:17.450946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:17.450946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:36.054828+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364246+0000","last_change":"2024-03-04T16:19:04.781178+0000","last_active":"2024-03-04T16:19:30.364246+0000","last_peered":"2024-03-04T16:19:30.364246+0000","last_clean":"2024-03-04T16:19:30.364246+0000","last_became_active":"2024-03-04T16:19:04.780845+0000","last_became_peered":"2024-03-04T16:19:04.780845+0000","last_unstale":"2024-03-04T16:19:30.364246+0000","last_undegraded":"2024-03-04T16:19:30.364246+0000","last_fullsized":"2024-03-04T16:19:30.364246+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:16:10.021439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":276,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.388750+0000","last_change":"2024-03-04T16:16:57.389891+0000","last_active":"2024-03-04T16:19:30.388750+0000","last_peered":"2024-03-04T16:19:30.388750+0000","last_clean":"2024-03-04T16:19:30.388750+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:19:30.388750+0000","last_undegraded":"2024-03-04T16:19:30.388750+0000","last_fullsized":"2024-03-04T16:19:30.388750+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:16:57.389758+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.389758+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367127+0000","last_change":"2024-03-04T16:19:04.756381+0000","last_active":"2024-03-04T16:19:30.367127+0000","last_peered":"2024-03-04T16:19:30.367127+0000","last_clean":"2024-03-04T16:19:30.367127+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:19:30.367127+0000","last_undegraded":"2024-03-04T16:19:30.367127+0000","last_fullsized":"2024-03-04T16:19:30.367127+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.696369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.696369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"351'11947","reported_seq":12071,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.870358+0000","last_change":"2024-03-04T16:16:53.666174+0000","last_active":"2024-03-04T16:19:33.870358+0000","last_peered":"2024-03-04T16:19:33.870358+0000","last_clean":"2024-03-04T16:19:33.870358+0000","last_became_active":"2024-03-04T16:16:53.665651+0000","last_became_peered":"2024-03-04T16:16:53.665651+0000","last_unstale":"2024-03-04T16:19:33.870358+0000","last_undegraded":"2024-03-04T16:19:33.870358+0000","last_fullsized":"2024-03-04T16:19:33.870358+0000","mapping_epoch":296,"log_start":"325'8180","ondisk_log_start":"325'8180","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310968320,"num_objects":4745,"num_object_clones":0,"num_object_copies":14235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11947,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":627,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368958+0000","last_change":"2024-03-04T16:17:46.756768+0000","last_active":"2024-03-04T16:19:30.368958+0000","last_peered":"2024-03-04T16:19:30.368958+0000","last_clean":"2024-03-04T16:19:30.368958+0000","last_became_active":"2024-03-04T16:02:49.794337+0000","last_became_peered":"2024-03-04T16:02:49.794337+0000","last_unstale":"2024-03-04T16:19:30.368958+0000","last_undegraded":"2024-03-04T16:19:30.368958+0000","last_fullsized":"2024-03-04T16:19:30.368958+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.756659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.756659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":948,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363640+0000","last_change":"2024-03-04T16:19:04.548298+0000","last_active":"2024-03-04T16:19:30.363640+0000","last_peered":"2024-03-04T16:19:30.363640+0000","last_clean":"2024-03-04T16:19:30.363640+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:19:30.363640+0000","last_undegraded":"2024-03-04T16:19:30.363640+0000","last_fullsized":"2024-03-04T16:19:30.363640+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:19:04.548119+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:19:04.548119+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:08.351610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381893+0000","last_change":"2024-03-04T16:16:20.104755+0000","last_active":"2024-03-04T16:19:30.381893+0000","last_peered":"2024-03-04T16:19:30.381893+0000","last_clean":"2024-03-04T16:19:30.381893+0000","last_became_active":"2024-03-04T16:10:29.339648+0000","last_became_peered":"2024-03-04T16:10:29.339648+0000","last_unstale":"2024-03-04T16:19:30.381893+0000","last_undegraded":"2024-03-04T16:19:30.381893+0000","last_fullsized":"2024-03-04T16:19:30.381893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.104643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.104643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"240'29","reported_seq":532,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367284+0000","last_change":"2024-03-04T16:17:05.303664+0000","last_active":"2024-03-04T16:19:30.367284+0000","last_peered":"2024-03-04T16:19:30.367284+0000","last_clean":"2024-03-04T16:19:30.367284+0000","last_became_active":"2024-03-04T16:10:29.322972+0000","last_became_peered":"2024-03-04T16:10:29.322972+0000","last_unstale":"2024-03-04T16:19:30.367284+0000","last_undegraded":"2024-03-04T16:19:30.367284+0000","last_fullsized":"2024-03-04T16:19:30.367284+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:17:05.303531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.303531+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":653,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364227+0000","last_change":"2024-03-04T16:19:04.750893+0000","last_active":"2024-03-04T16:19:30.364227+0000","last_peered":"2024-03-04T16:19:30.364227+0000","last_clean":"2024-03-04T16:19:30.364227+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:19:30.364227+0000","last_undegraded":"2024-03-04T16:19:30.364227+0000","last_fullsized":"2024-03-04T16:19:30.364227+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.429937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.429937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"351'9985","reported_seq":10118,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:33.853382+0000","last_change":"2024-03-04T16:19:05.512520+0000","last_active":"2024-03-04T16:19:33.853382+0000","last_peered":"2024-03-04T16:19:33.853382+0000","last_clean":"2024-03-04T16:19:03.686151+0000","last_became_active":"2024-03-04T16:19:05.495855+0000","last_became_peered":"2024-03-04T16:19:05.495855+0000","last_unstale":"2024-03-04T16:19:33.853382+0000","last_undegraded":"2024-03-04T16:19:33.853382+0000","last_fullsized":"2024-03-04T16:19:33.853382+0000","mapping_epoch":338,"log_start":"309'3767","ondisk_log_start":"309'3767","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":6218,"log_dups_size":0,"ondisk_log_size":6218,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161644544,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2466,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":276128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":2466}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":629,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.376965+0000","last_change":"2024-03-04T16:16:55.443627+0000","last_active":"2024-03-04T16:19:30.376965+0000","last_peered":"2024-03-04T16:19:30.376965+0000","last_clean":"2024-03-04T16:19:30.376965+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:19:30.376965+0000","last_undegraded":"2024-03-04T16:19:30.376965+0000","last_fullsized":"2024-03-04T16:19:30.376965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.443547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.443547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":484,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381969+0000","last_change":"2024-03-04T16:17:02.856208+0000","last_active":"2024-03-04T16:19:30.381969+0000","last_peered":"2024-03-04T16:19:30.381969+0000","last_clean":"2024-03-04T16:19:30.381969+0000","last_became_active":"2024-03-04T16:07:36.219856+0000","last_became_peered":"2024-03-04T16:07:36.219856+0000","last_unstale":"2024-03-04T16:19:30.381969+0000","last_undegraded":"2024-03-04T16:19:30.381969+0000","last_fullsized":"2024-03-04T16:19:30.381969+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:02.856124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:02.856124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1664,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:31.573635+0000","last_change":"2024-03-04T16:17:58.263035+0000","last_active":"2024-03-04T16:19:31.573635+0000","last_peered":"2024-03-04T16:19:31.573635+0000","last_clean":"2024-03-04T16:19:31.573635+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:19:31.573635+0000","last_undegraded":"2024-03-04T16:19:31.573635+0000","last_fullsized":"2024-03-04T16:19:31.573635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:17:58.262905+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.262905+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1002,"num_read_kb":997,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":548,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.389916+0000","last_change":"2024-03-04T16:19:04.893033+0000","last_active":"2024-03-04T16:19:30.389916+0000","last_peered":"2024-03-04T16:19:30.389916+0000","last_clean":"2024-03-04T16:19:30.389916+0000","last_became_active":"2024-03-04T16:19:04.826819+0000","last_became_peered":"2024-03-04T16:19:04.826819+0000","last_unstale":"2024-03-04T16:19:30.389916+0000","last_undegraded":"2024-03-04T16:19:30.389916+0000","last_fullsized":"2024-03-04T16:19:30.389916+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:53.588494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.588494+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369451+0000","last_change":"2024-03-04T16:17:43.601364+0000","last_active":"2024-03-04T16:19:30.369451+0000","last_peered":"2024-03-04T16:19:30.369451+0000","last_clean":"2024-03-04T16:19:30.369451+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:19:30.369451+0000","last_undegraded":"2024-03-04T16:19:30.369451+0000","last_fullsized":"2024-03-04T16:19:30.369451+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.600397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.600397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"351'12015","reported_seq":12140,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:32.820483+0000","last_change":"2024-03-04T16:16:53.661446+0000","last_active":"2024-03-04T16:19:32.820483+0000","last_peered":"2024-03-04T16:19:32.820483+0000","last_clean":"2024-03-04T16:19:32.820483+0000","last_became_active":"2024-03-04T16:16:53.661113+0000","last_became_peered":"2024-03-04T16:16:53.661113+0000","last_unstale":"2024-03-04T16:19:32.820483+0000","last_undegraded":"2024-03-04T16:19:32.820483+0000","last_fullsized":"2024-03-04T16:19:32.820483+0000","mapping_epoch":296,"log_start":"325'8305","ondisk_log_start":"325'8305","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320864280,"num_objects":4897,"num_object_clones":0,"num_object_copies":14691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12015,"num_write_kb":541121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":647,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.379443+0000","last_change":"2024-03-04T16:19:04.770031+0000","last_active":"2024-03-04T16:19:30.379443+0000","last_peered":"2024-03-04T16:19:30.379443+0000","last_clean":"2024-03-04T16:19:30.379443+0000","last_became_active":"2024-03-04T16:19:04.769649+0000","last_became_peered":"2024-03-04T16:19:04.769649+0000","last_unstale":"2024-03-04T16:19:30.379443+0000","last_undegraded":"2024-03-04T16:19:30.379443+0000","last_fullsized":"2024-03-04T16:19:30.379443+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_clean_scrub_stamp":"2024-03-04T16:18:09.376470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":645,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377477+0000","last_change":"2024-03-04T16:19:04.769538+0000","last_active":"2024-03-04T16:19:30.377477+0000","last_peered":"2024-03-04T16:19:30.377477+0000","last_clean":"2024-03-04T16:19:30.377477+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:19:30.377477+0000","last_undegraded":"2024-03-04T16:19:30.377477+0000","last_fullsized":"2024-03-04T16:19:30.377477+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:12.393386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:12.393386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:34.788670+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":614,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369920+0000","last_change":"2024-03-04T16:18:09.347836+0000","last_active":"2024-03-04T16:19:30.369920+0000","last_peered":"2024-03-04T16:19:30.369920+0000","last_clean":"2024-03-04T16:19:30.369920+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:19:30.369920+0000","last_undegraded":"2024-03-04T16:19:30.369920+0000","last_fullsized":"2024-03-04T16:19:30.369920+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:18:09.347708+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:18:09.347708+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":350,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.368476+0000","last_change":"2024-03-04T16:19:04.784739+0000","last_active":"2024-03-04T16:19:30.368476+0000","last_peered":"2024-03-04T16:19:30.368476+0000","last_clean":"2024-03-04T16:19:30.368476+0000","last_became_active":"2024-03-04T16:19:04.784531+0000","last_became_peered":"2024-03-04T16:19:04.784531+0000","last_unstale":"2024-03-04T16:19:30.368476+0000","last_undegraded":"2024-03-04T16:19:30.368476+0000","last_fullsized":"2024-03-04T16:19:30.368476+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:54.600256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:54.600256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"351'11853","reported_seq":12038,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:32.683086+0000","last_change":"2024-03-04T16:19:28.257681+0000","last_active":"2024-03-04T16:19:32.683086+0000","last_peered":"2024-03-04T16:19:32.683086+0000","last_clean":"2024-03-04T16:19:32.683086+0000","last_became_active":"2024-03-04T16:16:53.659725+0000","last_became_peered":"2024-03-04T16:16:53.659725+0000","last_unstale":"2024-03-04T16:19:32.683086+0000","last_undegraded":"2024-03-04T16:19:32.683086+0000","last_fullsized":"2024-03-04T16:19:32.683086+0000","mapping_epoch":296,"log_start":"318'7278","ondisk_log_start":"318'7278","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11595","last_scrub_stamp":"2024-03-04T16:19:28.257320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:19:28.257320+0000","objects_scrubbed":5199,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:31.138329+0000","scrub_duration":1649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320405504,"num_objects":4889,"num_object_clones":0,"num_object_copies":14667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11853,"num_write_kb":535744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":649,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381482+0000","last_change":"2024-03-04T16:19:04.849240+0000","last_active":"2024-03-04T16:19:30.381482+0000","last_peered":"2024-03-04T16:19:30.381482+0000","last_clean":"2024-03-04T16:19:30.381482+0000","last_became_active":"2024-03-04T16:19:04.834912+0000","last_became_peered":"2024-03-04T16:19:04.834912+0000","last_unstale":"2024-03-04T16:19:30.381482+0000","last_undegraded":"2024-03-04T16:19:30.381482+0000","last_fullsized":"2024-03-04T16:19:30.381482+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:05.755479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:05.755479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:30.642843+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":650,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370345+0000","last_change":"2024-03-04T16:19:04.763225+0000","last_active":"2024-03-04T16:19:30.370345+0000","last_peered":"2024-03-04T16:19:30.370345+0000","last_clean":"2024-03-04T16:19:30.370345+0000","last_became_active":"2024-03-04T16:19:04.762915+0000","last_became_peered":"2024-03-04T16:19:04.762915+0000","last_unstale":"2024-03-04T16:19:30.370345+0000","last_undegraded":"2024-03-04T16:19:30.370345+0000","last_fullsized":"2024-03-04T16:19:30.370345+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:01.874356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.874356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":672,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377475+0000","last_change":"2024-03-04T16:17:55.230783+0000","last_active":"2024-03-04T16:19:30.377475+0000","last_peered":"2024-03-04T16:19:30.377475+0000","last_clean":"2024-03-04T16:19:30.377475+0000","last_became_active":"2024-03-04T16:02:45.748595+0000","last_became_peered":"2024-03-04T16:02:45.748595+0000","last_unstale":"2024-03-04T16:19:30.377475+0000","last_undegraded":"2024-03-04T16:19:30.377475+0000","last_fullsized":"2024-03-04T16:19:30.377475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:17:55.230654+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.230654+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":482,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.371279+0000","last_change":"2024-03-04T16:16:11.811999+0000","last_active":"2024-03-04T16:19:30.371279+0000","last_peered":"2024-03-04T16:19:30.371279+0000","last_clean":"2024-03-04T16:19:30.371279+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:19:30.371279+0000","last_undegraded":"2024-03-04T16:19:30.371279+0000","last_fullsized":"2024-03-04T16:19:30.371279+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:16:11.811878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.811878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"351'12161","reported_seq":12295,"reported_epoch":351,"state":"active+remapped+backfill_wait","last_fresh":"2024-03-04T16:19:34.078417+0000","last_change":"2024-03-04T16:19:05.557431+0000","last_active":"2024-03-04T16:19:34.078417+0000","last_peered":"2024-03-04T16:19:34.078417+0000","last_clean":"2024-03-04T16:19:03.741183+0000","last_became_active":"2024-03-04T16:19:05.502251+0000","last_became_peered":"2024-03-04T16:19:05.502251+0000","last_unstale":"2024-03-04T16:19:34.078417+0000","last_undegraded":"2024-03-04T16:19:34.078417+0000","last_fullsized":"2024-03-04T16:19:34.078417+0000","mapping_epoch":338,"log_start":"324'8274","ondisk_log_start":"324'8274","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3887,"log_dups_size":0,"ondisk_log_size":3887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325124096,"num_objects":4961,"num_object_clones":0,"num_object_copies":14883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4961,"num_objects_unfound":0,"num_objects_dirty":4961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12161,"num_write_kb":547904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,4,1],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":4961}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":655,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.363954+0000","last_change":"2024-03-04T16:19:21.668870+0000","last_active":"2024-03-04T16:19:30.363954+0000","last_peered":"2024-03-04T16:19:30.363954+0000","last_clean":"2024-03-04T16:19:30.363954+0000","last_became_active":"2024-03-04T16:19:04.758780+0000","last_became_peered":"2024-03-04T16:19:04.758780+0000","last_unstale":"2024-03-04T16:19:30.363954+0000","last_undegraded":"2024-03-04T16:19:30.363954+0000","last_fullsized":"2024-03-04T16:19:30.363954+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:21.668734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:19:21.668734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:38.697802+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":480,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390034+0000","last_change":"2024-03-04T16:16:07.544810+0000","last_active":"2024-03-04T16:19:30.390034+0000","last_peered":"2024-03-04T16:19:30.390034+0000","last_clean":"2024-03-04T16:19:30.390034+0000","last_became_active":"2024-03-04T16:07:36.233936+0000","last_became_peered":"2024-03-04T16:07:36.233936+0000","last_unstale":"2024-03-04T16:19:30.390034+0000","last_undegraded":"2024-03-04T16:19:30.390034+0000","last_fullsized":"2024-03-04T16:19:30.390034+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:16:07.544679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.544679+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":700,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366492+0000","last_change":"2024-03-04T16:18:16.672174+0000","last_active":"2024-03-04T16:19:30.366492+0000","last_peered":"2024-03-04T16:19:30.366492+0000","last_clean":"2024-03-04T16:19:30.366492+0000","last_became_active":"2024-03-04T16:10:29.326009+0000","last_became_peered":"2024-03-04T16:10:29.326009+0000","last_unstale":"2024-03-04T16:19:30.366492+0000","last_undegraded":"2024-03-04T16:19:30.366492+0000","last_fullsized":"2024-03-04T16:19:30.366492+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'35","last_scrub_stamp":"2024-03-04T16:18:16.671642+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.671642+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"75'8","reported_seq":828,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381545+0000","last_change":"2024-03-04T16:17:06.654928+0000","last_active":"2024-03-04T16:19:30.381545+0000","last_peered":"2024-03-04T16:19:30.381545+0000","last_clean":"2024-03-04T16:19:30.381545+0000","last_became_active":"2024-03-04T16:02:47.767464+0000","last_became_peered":"2024-03-04T16:02:47.767464+0000","last_unstale":"2024-03-04T16:19:30.381545+0000","last_undegraded":"2024-03-04T16:19:30.381545+0000","last_fullsized":"2024-03-04T16:19:30.381545+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:06.654784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.654784+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"351'12077","reported_seq":16072,"reported_epoch":351,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:19:32.824660+0000","last_change":"2024-03-04T16:19:21.394566+0000","last_active":"2024-03-04T16:19:32.824660+0000","last_peered":"2024-03-04T16:19:32.824660+0000","last_clean":"2024-03-04T16:19:03.708860+0000","last_became_active":"2024-03-04T16:19:05.440750+0000","last_became_peered":"2024-03-04T16:19:05.440750+0000","last_unstale":"2024-03-04T16:19:32.824660+0000","last_undegraded":"2024-03-04T16:19:32.824660+0000","last_fullsized":"2024-03-04T16:19:32.824660+0000","mapping_epoch":338,"log_start":"325'8386","ondisk_log_start":"325'8386","created":296,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3691,"log_dups_size":0,"ondisk_log_size":3691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327876608,"num_objects":5003,"num_object_clones":0,"num_object_copies":15009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3178,"num_objects_unfound":0,"num_objects_dirty":5003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12077,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1931,"num_bytes_recovered":126550016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,5,3],"avail_no_missing":["4","3","5"],"object_location_counts":[{"shards":"3,4,5","objects":5003}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":639,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366736+0000","last_change":"2024-03-04T16:19:04.749049+0000","last_active":"2024-03-04T16:19:30.366736+0000","last_peered":"2024-03-04T16:19:30.366736+0000","last_clean":"2024-03-04T16:19:30.366736+0000","last_became_active":"2024-03-04T16:19:04.748656+0000","last_became_peered":"2024-03-04T16:19:04.748656+0000","last_unstale":"2024-03-04T16:19:30.366736+0000","last_undegraded":"2024-03-04T16:19:30.366736+0000","last_fullsized":"2024-03-04T16:19:30.366736+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:15.665346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.665346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":636,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.390182+0000","last_change":"2024-03-04T16:17:47.139524+0000","last_active":"2024-03-04T16:19:30.390182+0000","last_peered":"2024-03-04T16:19:30.390182+0000","last_clean":"2024-03-04T16:19:30.390182+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:19:30.390182+0000","last_undegraded":"2024-03-04T16:19:30.390182+0000","last_fullsized":"2024-03-04T16:19:30.390182+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.139403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.139403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":727,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381401+0000","last_change":"2024-03-04T16:18:07.422343+0000","last_active":"2024-03-04T16:19:30.381401+0000","last_peered":"2024-03-04T16:19:30.381401+0000","last_clean":"2024-03-04T16:19:30.381401+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:19:30.381401+0000","last_undegraded":"2024-03-04T16:19:30.381401+0000","last_fullsized":"2024-03-04T16:19:30.381401+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-03-04T16:18:07.422222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:07.422222+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":625,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366302+0000","last_change":"2024-03-04T16:17:45.302896+0000","last_active":"2024-03-04T16:19:30.366302+0000","last_peered":"2024-03-04T16:19:30.366302+0000","last_clean":"2024-03-04T16:19:30.366302+0000","last_became_active":"2024-03-04T16:02:47.766834+0000","last_became_peered":"2024-03-04T16:02:47.766834+0000","last_unstale":"2024-03-04T16:19:30.366302+0000","last_undegraded":"2024-03-04T16:19:30.366302+0000","last_fullsized":"2024-03-04T16:19:30.366302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:45.302787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:45.302787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"351'9659","reported_seq":9789,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:32.537274+0000","last_change":"2024-03-04T16:18:19.367339+0000","last_active":"2024-03-04T16:19:32.537274+0000","last_peered":"2024-03-04T16:19:32.537274+0000","last_clean":"2024-03-04T16:19:32.537274+0000","last_became_active":"2024-03-04T16:18:19.364862+0000","last_became_peered":"2024-03-04T16:18:19.364862+0000","last_unstale":"2024-03-04T16:19:32.537274+0000","last_undegraded":"2024-03-04T16:19:32.537274+0000","last_fullsized":"2024-03-04T16:19:32.537274+0000","mapping_epoch":296,"log_start":"312'4669","ondisk_log_start":"312'4669","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:35.806069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161579008,"num_objects":2465,"num_object_clones":0,"num_object_copies":7395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5969,"num_write_kb":269920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":631,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369945+0000","last_change":"2024-03-04T16:17:53.653243+0000","last_active":"2024-03-04T16:19:30.369945+0000","last_peered":"2024-03-04T16:19:30.369945+0000","last_clean":"2024-03-04T16:19:30.369945+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:19:30.369945+0000","last_undegraded":"2024-03-04T16:19:30.369945+0000","last_fullsized":"2024-03-04T16:19:30.369945+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:53.653131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.653131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"75'8","reported_seq":834,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370120+0000","last_change":"2024-03-04T16:17:58.494941+0000","last_active":"2024-03-04T16:19:30.370120+0000","last_peered":"2024-03-04T16:19:30.370120+0000","last_clean":"2024-03-04T16:19:30.370120+0000","last_became_active":"2024-03-04T16:02:47.768365+0000","last_became_peered":"2024-03-04T16:02:47.768365+0000","last_unstale":"2024-03-04T16:19:30.370120+0000","last_undegraded":"2024-03-04T16:19:30.370120+0000","last_fullsized":"2024-03-04T16:19:30.370120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:58.494829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.494829+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.381310+0000","last_change":"2024-03-04T16:17:44.202256+0000","last_active":"2024-03-04T16:19:30.381310+0000","last_peered":"2024-03-04T16:19:30.381310+0000","last_clean":"2024-03-04T16:19:30.381310+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:19:30.381310+0000","last_undegraded":"2024-03-04T16:19:30.381310+0000","last_fullsized":"2024-03-04T16:19:30.381310+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:44.202137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.202137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"301'34","reported_seq":678,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.389055+0000","last_change":"2024-03-04T16:17:05.452594+0000","last_active":"2024-03-04T16:19:30.389055+0000","last_peered":"2024-03-04T16:19:30.389055+0000","last_clean":"2024-03-04T16:19:30.389055+0000","last_became_active":"2024-03-04T16:02:45.745208+0000","last_became_peered":"2024-03-04T16:02:45.745208+0000","last_unstale":"2024-03-04T16:19:30.389055+0000","last_undegraded":"2024-03-04T16:19:30.389055+0000","last_fullsized":"2024-03-04T16:19:30.389055+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'34","last_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.452417+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"351'9814","reported_seq":9944,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.835053+0000","last_change":"2024-03-04T16:18:19.366813+0000","last_active":"2024-03-04T16:19:33.835053+0000","last_peered":"2024-03-04T16:19:33.835053+0000","last_clean":"2024-03-04T16:19:33.835053+0000","last_became_active":"2024-03-04T16:18:19.363858+0000","last_became_peered":"2024-03-04T16:18:19.363858+0000","last_unstale":"2024-03-04T16:19:33.835053+0000","last_undegraded":"2024-03-04T16:19:33.835053+0000","last_fullsized":"2024-03-04T16:19:33.835053+0000","mapping_epoch":296,"log_start":"312'4709","ondisk_log_start":"312'4709","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:34.753899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160563200,"num_objects":2450,"num_object_clones":0,"num_object_copies":7350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6088,"num_write_kb":273216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":623,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.383247+0000","last_change":"2024-03-04T16:19:04.549243+0000","last_active":"2024-03-04T16:19:30.383247+0000","last_peered":"2024-03-04T16:19:30.383247+0000","last_clean":"2024-03-04T16:19:30.383247+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:19:30.383247+0000","last_undegraded":"2024-03-04T16:19:30.383247+0000","last_fullsized":"2024-03-04T16:19:30.383247+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:19:04.549160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:19:04.549160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:24.214972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":635,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.369360+0000","last_change":"2024-03-04T16:19:04.757480+0000","last_active":"2024-03-04T16:19:30.369360+0000","last_peered":"2024-03-04T16:19:30.369360+0000","last_clean":"2024-03-04T16:19:30.369360+0000","last_became_active":"2024-03-04T16:19:04.757118+0000","last_became_peered":"2024-03-04T16:19:04.757118+0000","last_unstale":"2024-03-04T16:19:30.369360+0000","last_undegraded":"2024-03-04T16:19:30.369360+0000","last_fullsized":"2024-03-04T16:19:30.369360+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.806254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.806254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":636,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366397+0000","last_change":"2024-03-04T16:19:04.748781+0000","last_active":"2024-03-04T16:19:30.366397+0000","last_peered":"2024-03-04T16:19:30.366397+0000","last_clean":"2024-03-04T16:19:30.366397+0000","last_became_active":"2024-03-04T16:19:04.747992+0000","last_became_peered":"2024-03-04T16:19:04.747992+0000","last_unstale":"2024-03-04T16:19:30.366397+0000","last_undegraded":"2024-03-04T16:19:30.366397+0000","last_fullsized":"2024-03-04T16:19:30.366397+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:12.534849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:16:12.534849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":682,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367508+0000","last_change":"2024-03-04T16:17:00.348913+0000","last_active":"2024-03-04T16:19:30.367508+0000","last_peered":"2024-03-04T16:19:30.367508+0000","last_clean":"2024-03-04T16:19:30.367508+0000","last_became_active":"2024-03-04T16:10:29.330139+0000","last_became_peered":"2024-03-04T16:10:29.330139+0000","last_unstale":"2024-03-04T16:19:30.367508+0000","last_undegraded":"2024-03-04T16:19:30.367508+0000","last_fullsized":"2024-03-04T16:19:30.367508+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:00.348771+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.348771+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"351'9781","reported_seq":9917,"reported_epoch":351,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:19:33.916013+0000","last_change":"2024-03-04T16:19:05.515381+0000","last_active":"2024-03-04T16:19:33.916013+0000","last_peered":"2024-03-04T16:19:33.916013+0000","last_clean":"2024-03-04T16:19:03.720711+0000","last_became_active":"2024-03-04T16:19:05.490286+0000","last_became_peered":"2024-03-04T16:19:05.490286+0000","last_unstale":"2024-03-04T16:19:33.916013+0000","last_undegraded":"2024-03-04T16:19:33.916013+0000","last_fullsized":"2024-03-04T16:19:33.916013+0000","mapping_epoch":338,"log_start":"312'4474","ondisk_log_start":"312'4474","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5307,"log_dups_size":0,"ondisk_log_size":5307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156008448,"num_objects":2380,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2380,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6044,"num_write_kb":269600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[3,5,4],"avail_no_missing":["3","4","5"],"object_location_counts":[{"shards":"3,4,5","objects":2380}],"blocked_by":[],"up_primary":2,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":37,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.365622+0000","last_change":"2024-03-04T16:19:04.771692+0000","last_active":"2024-03-04T16:19:30.365622+0000","last_peered":"2024-03-04T16:19:30.365622+0000","last_clean":"2024-03-04T16:19:30.365622+0000","last_became_active":"2024-03-04T16:19:04.771353+0000","last_became_peered":"2024-03-04T16:19:04.771353+0000","last_unstale":"2024-03-04T16:19:30.365622+0000","last_undegraded":"2024-03-04T16:19:30.365622+0000","last_fullsized":"2024-03-04T16:19:30.365622+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.301767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.301767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.377681+0000","last_change":"2024-03-04T16:17:59.342404+0000","last_active":"2024-03-04T16:19:30.377681+0000","last_peered":"2024-03-04T16:19:30.377681+0000","last_clean":"2024-03-04T16:19:30.377681+0000","last_became_active":"2024-03-04T16:02:47.768143+0000","last_became_peered":"2024-03-04T16:02:47.768143+0000","last_unstale":"2024-03-04T16:19:30.377681+0000","last_undegraded":"2024-03-04T16:19:30.377681+0000","last_fullsized":"2024-03-04T16:19:30.377681+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:59.342270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:59.342270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":344,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366784+0000","last_change":"2024-03-04T16:16:02.539110+0000","last_active":"2024-03-04T16:19:30.366784+0000","last_peered":"2024-03-04T16:19:30.366784+0000","last_clean":"2024-03-04T16:19:30.366784+0000","last_became_active":"2024-03-04T16:10:29.321691+0000","last_became_peered":"2024-03-04T16:10:29.321691+0000","last_unstale":"2024-03-04T16:19:30.366784+0000","last_undegraded":"2024-03-04T16:19:30.366784+0000","last_fullsized":"2024-03-04T16:19:30.366784+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:02.538977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:16:02.538977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":749,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.367443+0000","last_change":"2024-03-04T16:19:04.854535+0000","last_active":"2024-03-04T16:19:30.367443+0000","last_peered":"2024-03-04T16:19:30.367443+0000","last_clean":"2024-03-04T16:19:30.367443+0000","last_became_active":"2024-03-04T16:19:04.777315+0000","last_became_peered":"2024-03-04T16:19:04.777315+0000","last_unstale":"2024-03-04T16:19:30.367443+0000","last_undegraded":"2024-03-04T16:19:30.367443+0000","last_fullsized":"2024-03-04T16:19:30.367443+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:17:01.304925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.304925+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"351'9874","reported_seq":10004,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:33.848424+0000","last_change":"2024-03-04T16:18:19.363300+0000","last_active":"2024-03-04T16:19:33.848424+0000","last_peered":"2024-03-04T16:19:33.848424+0000","last_clean":"2024-03-04T16:19:33.848424+0000","last_became_active":"2024-03-04T16:18:19.360752+0000","last_became_peered":"2024-03-04T16:18:19.360752+0000","last_unstale":"2024-03-04T16:19:33.848424+0000","last_undegraded":"2024-03-04T16:19:33.848424+0000","last_fullsized":"2024-03-04T16:19:33.848424+0000","mapping_epoch":296,"log_start":"312'4756","ondisk_log_start":"312'4756","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5118,"log_dups_size":0,"ondisk_log_size":5118,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:19:34.753926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162889728,"num_objects":2485,"num_object_clones":0,"num_object_copies":7455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6107,"num_write_kb":274976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":632,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.370115+0000","last_change":"2024-03-04T16:19:04.778715+0000","last_active":"2024-03-04T16:19:30.370115+0000","last_peered":"2024-03-04T16:19:30.370115+0000","last_clean":"2024-03-04T16:19:30.370115+0000","last_became_active":"2024-03-04T16:19:04.777606+0000","last_became_peered":"2024-03-04T16:19:04.777606+0000","last_unstale":"2024-03-04T16:19:30.370115+0000","last_undegraded":"2024-03-04T16:19:30.370115+0000","last_fullsized":"2024-03-04T16:19:30.370115+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:11.633381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:11.633381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":47,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.364842+0000","last_change":"2024-03-04T16:19:04.817708+0000","last_active":"2024-03-04T16:19:30.364842+0000","last_peered":"2024-03-04T16:19:30.364842+0000","last_clean":"2024-03-04T16:19:30.364842+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:19:30.364842+0000","last_undegraded":"2024-03-04T16:19:30.364842+0000","last_fullsized":"2024-03-04T16:19:30.364842+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:17:44.249367+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.249367+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":636,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.385159+0000","last_change":"2024-03-04T16:18:08.455945+0000","last_active":"2024-03-04T16:19:30.385159+0000","last_peered":"2024-03-04T16:19:30.385159+0000","last_clean":"2024-03-04T16:19:30.385159+0000","last_became_active":"2024-03-04T16:07:36.222708+0000","last_became_peered":"2024-03-04T16:07:36.222708+0000","last_unstale":"2024-03-04T16:19:30.385159+0000","last_undegraded":"2024-03-04T16:19:30.385159+0000","last_fullsized":"2024-03-04T16:19:30.385159+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:08.455830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:18:08.455830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"318'25","reported_seq":681,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-03-04T16:19:30.366938+0000","last_change":"2024-03-04T16:18:14.651077+0000","last_active":"2024-03-04T16:19:30.366938+0000","last_peered":"2024-03-04T16:19:30.366938+0000","last_clean":"2024-03-04T16:19:30.366938+0000","last_became_active":"2024-03-04T16:10:29.252691+0000","last_became_peered":"2024-03-04T16:10:29.252691+0000","last_unstale":"2024-03-04T16:19:30.366938+0000","last_undegraded":"2024-03-04T16:19:30.366938+0000","last_fullsized":"2024-03-04T16:19:30.366938+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'25","last_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.650758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":5169119256,"num_objects":78876,"num_object_clones":0,"num_object_copies":236628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":37520,"num_objects_unfound":0,"num_objects_dirty":78876,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":187284,"num_write_kb":8428129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37797,"num_bytes_recovered":2476867584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17745129472,"data_stored":17745117256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139917,"ondisk_log_size":139917,"up":78,"acting":78,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1002,"num_read_kb":997,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":5848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":755,"ondisk_log_size":755,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070670,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2589936,"kb_used_data":1886824,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91155216,"statfs":{"total":95995035648,"available":93342941184,"internally_reserved":0,"allocated":1932107776,"data_stored":1931561875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719976906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2967248,"kb_used_data":2069768,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":90777904,"statfs":{"total":95995035648,"available":92956573696,"internally_reserved":0,"allocated":2119442432,"data_stored":2118863210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738608,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4068288,"kb_used_data":3306872,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":89676864,"statfs":{"total":95995035648,"available":91829108736,"internally_reserved":0,"allocated":3386236928,"data_stored":3385674391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":991884,"kb_used_data":599748,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92753268,"statfs":{"total":95995035648,"available":94979346432,"internally_reserved":0,"allocated":614141952,"data_stored":613630546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149921,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3854980,"kb_used_data":2895036,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":89890172,"statfs":{"total":95995035648,"available":92047536128,"internally_reserved":0,"allocated":2964516864,"data_stored":2963949624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":982972872},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953692,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4801396,"kb_used_data":4124460,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":88943756,"statfs":{"total":95995035648,"available":91078406144,"internally_reserved":0,"allocated":4223447040,"data_stored":4222895298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626648,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3420580,"kb_used_data":2523032,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":90324572,"statfs":{"total":95995035648,"available":92492361728,"internally_reserved":0,"allocated":2583584768,"data_stored":2583053038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":919009734},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3351908352,"data_stored":3351904280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2114781184,"data_stored":2114781184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":614268928,"data_stored":614268928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2947289088,"data_stored":2947285016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4221308928,"data_stored":4221304856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2566651904,"data_stored":2566651904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1928921088,"data_stored":1928921088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:19:38.406 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-04T16:19:38.416 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.449 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:38.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:38.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:38.591 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:38.693 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:38.717 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:38.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[29779]: from='client.25675 v1:172.21.15.38:0/2450634638' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[29779]: osdmap e353: 8 total, 8 up, 7 in 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[29779]: pgmap v764: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 701 B/s rd, 0 B/s wr, 573 op/s; 37520/237222 objects misplaced (15.816%); 16 MiB/s, 257 objects/s recovering 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[38113]: from='client.25675 v1:172.21.15.38:0/2450634638' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[38113]: osdmap e353: 8 total, 8 up, 7 in 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[38113]: pgmap v764: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 701 B/s rd, 0 B/s wr, 573 op/s; 37520/237222 objects misplaced (15.816%); 16 MiB/s, 257 objects/s recovering 2024-03-04T16:19:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:38.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:38 smithi176 ceph-mon[33619]: from='client.25675 v1:172.21.15.38:0/2450634638' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:19:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:38 smithi176 ceph-mon[33619]: osdmap e353: 8 total, 8 up, 7 in 2024-03-04T16:19:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:38 smithi176 ceph-mon[33619]: pgmap v764: 158 pgs: 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_wait, 145 active+clean, 4 active+remapped+backfilling, 2 active+remapped+backfill_toofull; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 701 B/s rd, 0 B/s wr, 573 op/s; 37520/237222 objects misplaced (15.816%); 16 MiB/s, 257 objects/s recovering 2024-03-04T16:19:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:38.859 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:38.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:38.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:38.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:39.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:39.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:39.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:39.316 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:39.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:39.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:39.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:39.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:39.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:39.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:39.729 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:39.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:39.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:39.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:39.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.015 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:40.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:40.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:39 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:40.116 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:40.128 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:39 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:40.128 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:39 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:19:40.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:40.267 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:40.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:40.337 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:40.358 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:19:40.358 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:19:40.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:40.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:40.559 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:40.594 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:40.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:40.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:40.786 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:19:40.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:40.863 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:40.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:40.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:40.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[29779]: pgmap v765: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 325 op/s; 32027/224178 objects misplaced (14.286%); 5.0 MiB/s, 79 objects/s recovering 2024-03-04T16:19:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3594369325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[29779]: from='client.25681 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[38113]: pgmap v765: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 325 op/s; 32027/224178 objects misplaced (14.286%); 5.0 MiB/s, 79 objects/s recovering 2024-03-04T16:19:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3594369325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:40 smithi038 ceph-mon[38113]: from='client.25681 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:41.001 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:41.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:41.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:40 smithi176 ceph-mon[33619]: pgmap v765: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 325 op/s; 32027/224178 objects misplaced (14.286%); 5.0 MiB/s, 79 objects/s recovering 2024-03-04T16:19:41.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:40 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3594369325' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:41.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:40 smithi176 ceph-mon[33619]: from='client.25681 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:19:41.135 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:41.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:41.279 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:41.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:41.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:41.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:41.566 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:41.668 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:41.699 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:41.801 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:41.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:41.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:41.980 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:41.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:42.027 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:19:42.070 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:19:42.070 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[38113]: from='client.25681 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:19:42.070 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[38113]: osdmap e354: 8 total, 8 up, 7 in 2024-03-04T16:19:42.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:19:42.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[29779]: from='client.25681 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:19:42.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:41 smithi038 ceph-mon[29779]: osdmap e354: 8 total, 8 up, 7 in 2024-03-04T16:19:42.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:41 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:19:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:41 smithi176 ceph-mon[33619]: from='client.25681 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:19:42.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:41 smithi176 ceph-mon[33619]: osdmap e354: 8 total, 8 up, 7 in 2024-03-04T16:19:42.110 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:42.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:42.266 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:42.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:42.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:42.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:42.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:42.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.709 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:42.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:42.810 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:42.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:42.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:42.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:42.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:43.091 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:43.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:42 smithi176 ceph-mon[33619]: pgmap v767: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 32027/224178 objects misplaced (14.286%) 2024-03-04T16:19:43.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:42 smithi038 ceph-mon[29779]: pgmap v767: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 32027/224178 objects misplaced (14.286%) 2024-03-04T16:19:43.123 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:43.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:42 smithi038 ceph-mon[38113]: pgmap v767: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 32027/224178 objects misplaced (14.286%) 2024-03-04T16:19:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:43.225 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:43.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:43.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:43.408 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 7, 2] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:19:43.408 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:19:43.408 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 5, 7, 2] 2024-03-04T16:19:43.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd out 4 2024-03-04T16:19:43.439 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:43.456 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:43.540 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:43.570 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:43.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:43.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:43.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:43.702 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:43.714 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:43.804 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:43.854 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:43.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:43.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:44.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:44.040 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:44.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:44.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:44.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:44.156 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:44.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:44.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:44.298 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:44.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:44.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.441 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:44.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:44.532 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:19:44.542 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:44.569 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:44.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:44.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:44.802 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:44.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:19:44.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:19:44.830 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[29779]: pgmap v768: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 100 op/s; 30029/217260 objects misplaced (13.822%) 2024-03-04T16:19:44.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[29779]: osdmap e355: 8 total, 8 up, 7 in 2024-03-04T16:19:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:44.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:44.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.007 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:44 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:19:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:44 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:19:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:44 smithi176 ceph-mon[33619]: pgmap v768: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 100 op/s; 30029/217260 objects misplaced (13.822%) 2024-03-04T16:19:45.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:44 smithi176 ceph-mon[33619]: osdmap e355: 8 total, 8 up, 7 in 2024-03-04T16:19:45.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:45.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:19:45.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:19:45.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[38113]: pgmap v768: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.4 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 100 op/s; 30029/217260 objects misplaced (13.822%) 2024-03-04T16:19:45.111 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:44 smithi038 ceph-mon[38113]: osdmap e355: 8 total, 8 up, 7 in 2024-03-04T16:19:45.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:45.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.274 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:45.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:45.422 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:45.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:45.553 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:45.655 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:45.688 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:45.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:45.818 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:45.921 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:45.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:45.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:46.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:46.103 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:46.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:46.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:46.335 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:46.377 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:46.379 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:19:46.379 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:19:46.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:46.514 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:46.615 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:46.645 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2574239101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2574239101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:19:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:46 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:19:46.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:46.778 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2574239101' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:19:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:46 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:19:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:46 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:19:46.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:46.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:46.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:47.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:47.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:47.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:47.204 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:47.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:47.337 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:47.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:19:47.446 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:19:47.447 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:19:47.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:47.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:47.620 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:47.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:47 smithi038 ceph-mon[29779]: pgmap v770: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 26929/208566 objects misplaced (12.912%); 4.8 MiB/s, 76 objects/s recovering 2024-03-04T16:19:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:47 smithi038 ceph-mon[29779]: osdmap e356: 8 total, 8 up, 7 in 2024-03-04T16:19:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:47 smithi038 ceph-mon[38113]: pgmap v770: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 26929/208566 objects misplaced (12.912%); 4.8 MiB/s, 76 objects/s recovering 2024-03-04T16:19:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:47 smithi038 ceph-mon[38113]: osdmap e356: 8 total, 8 up, 7 in 2024-03-04T16:19:47.762 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:47.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:19:47.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:47 smithi176 ceph-mon[33619]: pgmap v770: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 26929/208566 objects misplaced (12.912%); 4.8 MiB/s, 76 objects/s recovering 2024-03-04T16:19:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:47 smithi176 ceph-mon[33619]: osdmap e356: 8 total, 8 up, 7 in 2024-03-04T16:19:47.864 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:47.936 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:47.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:48.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:48.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:48.124 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:48.171 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:48.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:48.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:48.336 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:48.438 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:48.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:48.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:48 smithi038 ceph-mon[29779]: pgmap v772: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 744 B/s rd, 0 B/s wr, 757 op/s; 26929/208566 objects misplaced (12.912%); 17 MiB/s, 269 objects/s recovering 2024-03-04T16:19:48.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:48 smithi038 ceph-mon[38113]: pgmap v772: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 744 B/s rd, 0 B/s wr, 757 op/s; 26929/208566 objects misplaced (12.912%); 17 MiB/s, 269 objects/s recovering 2024-03-04T16:19:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:48.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:48.637 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:48.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:48.745 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:19:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:48.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:48.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:48 smithi176 ceph-mon[33619]: pgmap v772: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 4.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 744 B/s rd, 0 B/s wr, 757 op/s; 26929/208566 objects misplaced (12.912%); 17 MiB/s, 269 objects/s recovering 2024-03-04T16:19:48.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:48.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:48.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:49.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:49.090 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:49.192 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:49.223 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:49.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:49.367 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:49.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:49.502 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:49.604 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:49.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:49.740 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:49.776 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:49.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:49.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:49.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:50.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:50.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:50.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:50.172 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:50.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:50.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:50.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:50.430 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:50.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:50.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:50.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:50.697 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:50.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:19:50.799 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:50.827 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:50.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:50.834 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:50 smithi038 ceph-mon[38113]: osdmap e357: 8 total, 8 up, 7 in 2024-03-04T16:19:50.834 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:50 smithi038 ceph-mon[38113]: pgmap v774: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 22178/193311 objects misplaced (11.473%); 26 MiB/s, 416 objects/s recovering 2024-03-04T16:19:50.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:50 smithi038 ceph-mon[29779]: osdmap e357: 8 total, 8 up, 7 in 2024-03-04T16:19:50.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:50 smithi038 ceph-mon[29779]: pgmap v774: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 22178/193311 objects misplaced (11.473%); 26 MiB/s, 416 objects/s recovering 2024-03-04T16:19:50.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:50 smithi176 ceph-mon[33619]: osdmap e357: 8 total, 8 up, 7 in 2024-03-04T16:19:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:50 smithi176 ceph-mon[33619]: pgmap v774: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s; 22178/193311 objects misplaced (11.473%); 26 MiB/s, 416 objects/s recovering 2024-03-04T16:19:50.929 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:50.954 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:50.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:51.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:51.079 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:51.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:51.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:51.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:51.344 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:51.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:51.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:51.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:51.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:51.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:51.612 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:51.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:51.746 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:51.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:51.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:51 smithi176 ceph-mon[33619]: osdmap e358: 8 total, 8 up, 7 in 2024-03-04T16:19:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:51.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:51.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:51.974 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:51.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:51 smithi038 ceph-mon[29779]: osdmap e358: 8 total, 8 up, 7 in 2024-03-04T16:19:51.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:51 smithi038 ceph-mon[38113]: osdmap e358: 8 total, 8 up, 7 in 2024-03-04T16:19:52.005 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:52.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:52.139 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:52.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:52.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:52.371 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:52.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:52.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:52.548 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:52.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:52.679 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:52.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:52.813 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:52 smithi176 ceph-mon[33619]: osdmap e359: 8 total, 8 up, 7 in 2024-03-04T16:19:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:52 smithi176 ceph-mon[33619]: pgmap v777: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 972 B/s rd, 0 B/s wr, 966 op/s; 22178/193311 objects misplaced (11.473%); 22 MiB/s, 346 objects/s recovering 2024-03-04T16:19:52.915 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:52.945 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:52.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:52.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:52 smithi038 ceph-mon[29779]: osdmap e359: 8 total, 8 up, 7 in 2024-03-04T16:19:52.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:52 smithi038 ceph-mon[29779]: pgmap v777: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 972 B/s rd, 0 B/s wr, 966 op/s; 22178/193311 objects misplaced (11.473%); 22 MiB/s, 346 objects/s recovering 2024-03-04T16:19:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:52 smithi038 ceph-mon[38113]: osdmap e359: 8 total, 8 up, 7 in 2024-03-04T16:19:52.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:52 smithi038 ceph-mon[38113]: pgmap v777: 158 pgs: 4 active+remapped+backfill_wait, 147 active+clean, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 972 B/s rd, 0 B/s wr, 966 op/s; 22178/193311 objects misplaced (11.473%); 22 MiB/s, 346 objects/s recovering 2024-03-04T16:19:53.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:53.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:53.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:53.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:53.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:53.341 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:53.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:53.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:53.481 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:53.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:19:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:19:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:19:53.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:53.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:19:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:19:53.614 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:53.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:53.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:53.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:53.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:53.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:53.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:54.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:54.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:54.160 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:54.262 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:54.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:54.401 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:54.435 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:54.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:54.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:54.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:54.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:54.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:54 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:19:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:19:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:19:54.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.718 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:54.819 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:54.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:54.950 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:54.978 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:54.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:55.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:55.110 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:55.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:55.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:55.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:55.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:55.406 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:55.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:55.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:55 smithi038 ceph-mon[29779]: pgmap v778: 158 pgs: 4 active+remapped+backfill_wait, 148 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s; 21569/188643 objects misplaced (11.434%); 25 MiB/s, 402 objects/s recovering 2024-03-04T16:19:55.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:55 smithi038 ceph-mon[38113]: pgmap v778: 158 pgs: 4 active+remapped+backfill_wait, 148 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s; 21569/188643 objects misplaced (11.434%); 25 MiB/s, 402 objects/s recovering 2024-03-04T16:19:55.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:55.539 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:55.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:55 smithi176 ceph-mon[33619]: pgmap v778: 158 pgs: 4 active+remapped+backfill_wait, 148 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.11k op/s; 21569/188643 objects misplaced (11.434%); 25 MiB/s, 402 objects/s recovering 2024-03-04T16:19:55.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.665 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:55.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:55.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:55.798 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:55.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:55.938 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:55.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:56.040 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:56.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:56.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:56.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:56.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:56.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:56.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:56.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:56.572 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:56.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:56.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:56.711 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:56.813 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:56.838 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:56.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:56.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:56.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:57.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:57.100 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:57.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:57.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.252 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:57.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:57.353 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:57.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:57 smithi038 ceph-mon[29779]: pgmap v779: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 839 B/s rd, 0 B/s wr, 645 op/s; 16808/181521 objects misplaced (9.260%); 28 MiB/s, 445 objects/s recovering 2024-03-04T16:19:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:57 smithi038 ceph-mon[38113]: pgmap v779: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 839 B/s rd, 0 B/s wr, 645 op/s; 16808/181521 objects misplaced (9.260%); 28 MiB/s, 445 objects/s recovering 2024-03-04T16:19:57.496 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:57.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:57.607 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:57 smithi176 ceph-mon[33619]: pgmap v779: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 839 B/s rd, 0 B/s wr, 645 op/s; 16808/181521 objects misplaced (9.260%); 28 MiB/s, 445 objects/s recovering 2024-03-04T16:19:57.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:57.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:57.760 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:57.905 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:57.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:57.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:58.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:58.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:58.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:58.183 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:58.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:58.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:58.346 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:58.447 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:58.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:58.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:58.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:19:58.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:58.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:19:58.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:58.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:58.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:58.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:19:58.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:58.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:19:59.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:19:59.039 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:59.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:19:59.140 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:19:59.169 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:19:59.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:19:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:59.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:19:59.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:59.461 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:59.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:19:59.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:19:59 smithi176 ceph-mon[33619]: pgmap v780: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 491 op/s; 16808/181521 objects misplaced (9.260%); 21 MiB/s, 339 objects/s recovering 2024-03-04T16:19:59.608 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:19:59.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:19:59 smithi038 ceph-mon[29779]: pgmap v780: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 491 op/s; 16808/181521 objects misplaced (9.260%); 21 MiB/s, 339 objects/s recovering 2024-03-04T16:19:59.696 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:19:59 smithi038 ceph-mon[38113]: pgmap v780: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 491 op/s; 16808/181521 objects misplaced (9.260%); 21 MiB/s, 339 objects/s recovering 2024-03-04T16:19:59.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:19:59.751 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:19:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:19:59.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:19:59.882 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:19:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:19:59.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:19:59.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:00.018 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:00.120 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:00.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:00.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: osdmap e360: 8 total, 8 up, 7 in 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: pg 10.3 is active+remapped+backfill_toofull, acting [3,5,4] 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: pg 10.12 is active+remapped+backfill_toofull, acting [0,1,4] 2024-03-04T16:20:00.287 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[29779]: pg 10.18 is active+remapped+backfill_toofull, acting [0,5,7] 2024-03-04T16:20:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.308 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:00.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:00.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:00.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.442 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:00.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:00.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:00.580 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: osdmap e360: 8 total, 8 up, 7 in 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: pg 10.3 is active+remapped+backfill_toofull, acting [3,5,4] 2024-03-04T16:20:00.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: pg 10.12 is active+remapped+backfill_toofull, acting [0,1,4] 2024-03-04T16:20:00.609 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:00 smithi176 ceph-mon[33619]: pg 10.18 is active+remapped+backfill_toofull, acting [0,5,7] 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: osdmap e360: 8 total, 8 up, 7 in 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-04T16:20:00.662 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-04T16:20:00.663 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: pg 10.3 is active+remapped+backfill_toofull, acting [3,5,4] 2024-03-04T16:20:00.663 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: pg 10.12 is active+remapped+backfill_toofull, acting [0,1,4] 2024-03-04T16:20:00.663 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:00 smithi038 ceph-mon[38113]: pg 10.18 is active+remapped+backfill_toofull, acting [0,5,7] 2024-03-04T16:20:00.673 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:00.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:00.759 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:00.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:00.911 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:00.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:01.012 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:01.062 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:01.164 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:01.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:01.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:01.332 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:01.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:01.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:01.617 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:01.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:01 smithi038 ceph-mon[29779]: pgmap v782: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 13766/166431 objects misplaced (8.271%); 29 MiB/s, 469 objects/s recovering 2024-03-04T16:20:01.644 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:01 smithi038 ceph-mon[29779]: osdmap e361: 8 total, 8 up, 7 in 2024-03-04T16:20:01.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:01 smithi038 ceph-mon[38113]: pgmap v782: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 13766/166431 objects misplaced (8.271%); 29 MiB/s, 469 objects/s recovering 2024-03-04T16:20:01.644 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:01 smithi038 ceph-mon[38113]: osdmap e361: 8 total, 8 up, 7 in 2024-03-04T16:20:01.719 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:01.738 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:01 smithi176 ceph-mon[33619]: pgmap v782: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 13766/166431 objects misplaced (8.271%); 29 MiB/s, 469 objects/s recovering 2024-03-04T16:20:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:01 smithi176 ceph-mon[33619]: osdmap e361: 8 total, 8 up, 7 in 2024-03-04T16:20:01.889 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:01.941 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:01.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:02.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:02.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:02.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:02.220 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:02.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:02.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:02.454 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:02.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:02.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:02.621 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:02.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:02.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:02.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:02.906 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:02.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:03.008 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:03.038 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:03.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:03.168 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:03.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:03.302 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:03.387 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:03.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:03.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:03.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.566 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:03.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:03.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:03 smithi176 ceph-mon[33619]: pgmap v784: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s; 13766/166431 objects misplaced (8.271%); 25 MiB/s, 400 objects/s recovering 2024-03-04T16:20:03.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:03.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:03.693 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:03.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:03 smithi038 ceph-mon[29779]: pgmap v784: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s; 13766/166431 objects misplaced (8.271%); 25 MiB/s, 400 objects/s recovering 2024-03-04T16:20:03.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:03 smithi038 ceph-mon[38113]: pgmap v784: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 925 op/s; 13766/166431 objects misplaced (8.271%); 25 MiB/s, 400 objects/s recovering 2024-03-04T16:20:03.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:03.830 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:03.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:03.967 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:03.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:04.081 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:04.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.107 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:04.208 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:04.238 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:04.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:04.374 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:04.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:04.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.509 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:04.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:04.642 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[29779]: osdmap e362: 8 total, 8 up, 7 in 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[29779]: pgmap v786: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 13766/160134 objects misplaced (8.597%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[38113]: osdmap e362: 8 total, 8 up, 7 in 2024-03-04T16:20:04.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:04 smithi038 ceph-mon[38113]: pgmap v786: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 13766/160134 objects misplaced (8.597%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:20:04.744 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:04.771 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:04 smithi176 ceph-mon[33619]: osdmap e362: 8 total, 8 up, 7 in 2024-03-04T16:20:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:04 smithi176 ceph-mon[33619]: pgmap v786: 158 pgs: 3 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s; 13766/160134 objects misplaced (8.597%); 11 MiB/s, 180 objects/s recovering 2024-03-04T16:20:04.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:04.897 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:04.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:04.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:05.036 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:05.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:05.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:05.179 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:05.281 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:05.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:05.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:05.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:05.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:05.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:05.588 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:05.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:05.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:05.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:05 smithi038 ceph-mon[29779]: osdmap e363: 8 total, 8 up, 7 in 2024-03-04T16:20:05.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:05 smithi038 ceph-mon[38113]: osdmap e363: 8 total, 8 up, 7 in 2024-03-04T16:20:05.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:05.854 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:05 smithi176 ceph-mon[33619]: osdmap e363: 8 total, 8 up, 7 in 2024-03-04T16:20:05.956 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:05.987 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:05.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:06.089 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:06.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:06.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:06.250 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:06.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:06.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:06.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:06.521 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:06.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:06.651 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:06.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[38113]: pgmap v788: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 10705/151149 objects misplaced (7.082%); 15 MiB/s, 234 objects/s recovering 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2910885385' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[38113]: from='client.25693 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[38113]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[29779]: pgmap v788: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 10705/151149 objects misplaced (7.082%); 15 MiB/s, 234 objects/s recovering 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:20:06.317+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e363 definitely_dead 0 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2910885385' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[29779]: from='client.25693 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:06 smithi038 ceph-mon[29779]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-04T16:20:06.753 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:06.776 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:06 smithi176 ceph-mon[33619]: pgmap v788: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 849 op/s; 10705/151149 objects misplaced (7.082%); 15 MiB/s, 234 objects/s recovering 2024-03-04T16:20:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:06 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2910885385' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:06 smithi176 ceph-mon[33619]: from='client.25693 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-04T16:20:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:06 smithi176 ceph-mon[33619]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-04T16:20:06.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:06.889 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.4. 2024-03-04T16:20:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:06.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:07.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:07.032 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:07.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:07.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.160 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:07.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:07.261 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:07.289 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:07.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:07.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:07.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:07.552 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:07.654 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:07.677 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:07.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:07 smithi038 ceph-mon[29779]: from='client.25693 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-04T16:20:07.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:07 smithi038 ceph-mon[29779]: osdmap e364: 8 total, 8 up, 6 in 2024-03-04T16:20:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:07 smithi038 ceph-mon[38113]: from='client.25693 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-04T16:20:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:07 smithi038 ceph-mon[38113]: osdmap e364: 8 total, 8 up, 6 in 2024-03-04T16:20:07.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:07.803 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:07 smithi176 ceph-mon[33619]: from='client.25693 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-04T16:20:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:07 smithi176 ceph-mon[33619]: osdmap e364: 8 total, 8 up, 6 in 2024-03-04T16:20:07.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:07.930 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:07.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:08.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.056 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:08.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:08.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:08.181 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:08.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:08.307 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:08.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:08.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:08.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:08.559 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:08.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:08.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[29779]: osdmap e365: 8 total, 8 up, 6 in 2024-03-04T16:20:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[29779]: pgmap v791: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 698 op/s; 10705/151149 objects misplaced (7.082%); 20 MiB/s, 327 objects/s recovering 2024-03-04T16:20:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[38113]: osdmap e365: 8 total, 8 up, 6 in 2024-03-04T16:20:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[38113]: pgmap v791: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 698 op/s; 10705/151149 objects misplaced (7.082%); 20 MiB/s, 327 objects/s recovering 2024-03-04T16:20:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:08.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.843 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:08.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:08 smithi176 ceph-mon[33619]: osdmap e365: 8 total, 8 up, 6 in 2024-03-04T16:20:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:08 smithi176 ceph-mon[33619]: pgmap v791: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 149 active+clean, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 0 B/s wr, 698 op/s; 10705/151149 objects misplaced (7.082%); 20 MiB/s, 327 objects/s recovering 2024-03-04T16:20:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:08.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:08.968 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:08.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:09.069 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:09.106 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:09.207 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:09.239 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:09.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:09.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:09.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:09.512 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:09.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:09.645 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.10 starting backfill to osd.5 from (0'0,0'0] MAX to 363'10601 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10810 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10767 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10641 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 363'13781 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10774 2024-03-04T16:20:09.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10847 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10889 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 363'13841 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: osdmap e366: 8 total, 8 up, 6 in 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10683 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 363'14088 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10702 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.10 starting backfill to osd.5 from (0'0,0'0] MAX to 363'10601 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10810 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10767 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10641 2024-03-04T16:20:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 363'13781 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10774 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10847 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10889 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 363'13841 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: osdmap e366: 8 total, 8 up, 6 in 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10683 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 363'14088 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10702 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:09.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:09.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:09.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.10 starting backfill to osd.5 from (0'0,0'0] MAX to 363'10601 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10810 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10767 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10641 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 363'13781 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.0 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10774 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10847 2024-03-04T16:20:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.19 starting backfill to osd.0 from (0'0,0'0] MAX to 363'10889 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.b starting backfill to osd.7 from (0'0,0'0] MAX to 363'13841 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: osdmap e366: 8 total, 8 up, 6 in 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 363'10683 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 363'14088 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 363'10702 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:09.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:09.880 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:09.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:09.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:10.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:10.059 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:10.161 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:10.192 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:10.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:10.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:10.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:10.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:10.570 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:10.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:10.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:10.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:10.790 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[29779]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10677 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[29779]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 363'13878 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[29779]: osdmap e367: 8 total, 8 up, 6 in 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[29779]: pgmap v794: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[38113]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10677 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[38113]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 363'13878 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[38113]: osdmap e367: 8 total, 8 up, 6 in 2024-03-04T16:20:10.791 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:10 smithi038 ceph-mon[38113]: pgmap v794: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:10.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:10.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:10 smithi176 ceph-mon[33619]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 363'10677 2024-03-04T16:20:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:10 smithi176 ceph-mon[33619]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 363'13878 2024-03-04T16:20:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:10 smithi176 ceph-mon[33619]: osdmap e367: 8 total, 8 up, 6 in 2024-03-04T16:20:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:10 smithi176 ceph-mon[33619]: pgmap v794: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:10.866 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:10.967 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:10.997 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:10.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:11.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:11.124 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:11.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:11.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:11.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:11.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:11.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:11.508 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:20:11.519 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:11.621 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:11.654 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:11.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:11.784 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:11.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:11.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:11 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:11 smithi176 ceph-mon[33619]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-04T16:20:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:11 smithi176 ceph-mon[33619]: osdmap e368: 8 total, 8 up, 6 in 2024-03-04T16:20:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2784325947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:11.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:11 smithi176 ceph-mon[33619]: from='client.25699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:11.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:11.928 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:11.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:11.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[29779]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[29779]: osdmap e368: 8 total, 8 up, 6 in 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2784325947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[29779]: from='client.25699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[38113]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[38113]: osdmap e368: 8 total, 8 up, 6 in 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2784325947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:11 smithi038 ceph-mon[38113]: from='client.25699 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:12.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:12.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:12.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:12.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:12.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:12.304 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:12.333 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:12.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:12.580 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:12.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:12.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:12.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:12.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: from='client.25699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: osdmap e369: 8 total, 8 up, 6 in 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: pgmap v797: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:12 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:12.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:12.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:12.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:12.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: from='client.25699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: osdmap e369: 8 total, 8 up, 6 in 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: pgmap v797: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: from='client.25699 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: osdmap e369: 8 total, 8 up, 6 in 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: pgmap v797: 158 pgs: 6 remapped+peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 138 active+clean, 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull; 2.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 516 op/s; 23229/144948 objects misplaced (16.026%); 33 MiB/s, 555 objects/s recovering 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:12 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.015 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:13.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:13.142 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:13.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:13.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.267 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:13.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:13.394 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:13.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:13.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:13 smithi176 ceph-mon[33619]: osdmap e370: 8 total, 8 up, 6 in 2024-03-04T16:20:13.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:13.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:13.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:13.654 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:13.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:13 smithi038 ceph-mon[29779]: osdmap e370: 8 total, 8 up, 6 in 2024-03-04T16:20:13.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:13 smithi038 ceph-mon[38113]: osdmap e370: 8 total, 8 up, 6 in 2024-03-04T16:20:13.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:13.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:13.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:13.787 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:13.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:13.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:14.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:14.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:14.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:14.197 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:14.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.394 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:14.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:14.495 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:14.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:14.620 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:14.651 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:14.753 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:14.777 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:14.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:14 smithi176 ceph-mon[33619]: osdmap e371: 8 total, 8 up, 6 in 2024-03-04T16:20:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:14 smithi176 ceph-mon[33619]: pgmap v800: 158 pgs: 1 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 139 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s; 31381/142272 objects misplaced (22.057%); 50 MiB/s, 797 objects/s recovering 2024-03-04T16:20:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:14 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:14 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:14.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:14.922 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:14.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:14.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[29779]: osdmap e371: 8 total, 8 up, 6 in 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[29779]: pgmap v800: 158 pgs: 1 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 139 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s; 31381/142272 objects misplaced (22.057%); 50 MiB/s, 797 objects/s recovering 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[38113]: osdmap e371: 8 total, 8 up, 6 in 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[38113]: pgmap v800: 158 pgs: 1 peering, 3 active+remapped+backfill_wait, 1 active+recovery_wait+degraded+remapped, 139 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 224 op/s; 31381/142272 objects misplaced (22.057%); 50 MiB/s, 797 objects/s recovering 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:14 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:15.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:15.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:15.154 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:15.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:15.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:15.316 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:15.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:15.447 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:15.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:15.577 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:15.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:15.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:15.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:15 smithi038 ceph-mon[29779]: osdmap e372: 8 total, 8 up, 6 in 2024-03-04T16:20:15.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:15 smithi038 ceph-mon[29779]: osdmap e373: 8 total, 8 up, 6 in 2024-03-04T16:20:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:15 smithi038 ceph-mon[38113]: osdmap e372: 8 total, 8 up, 6 in 2024-03-04T16:20:15.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:15 smithi038 ceph-mon[38113]: osdmap e373: 8 total, 8 up, 6 in 2024-03-04T16:20:15.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:15.839 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:15.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:15.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:15 smithi176 ceph-mon[33619]: osdmap e372: 8 total, 8 up, 6 in 2024-03-04T16:20:15.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:15 smithi176 ceph-mon[33619]: osdmap e373: 8 total, 8 up, 6 in 2024-03-04T16:20:15.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:15.975 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:15.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:16.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:16.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:16.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:16.236 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:16.338 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:16.363 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:16.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:16.501 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:16.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:16.633 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:16.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.784 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:16.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:16.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:16.915 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:16.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:17.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:17.055 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:17.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:17.194 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:17.296 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:17.325 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:17.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:17.427 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:17.456 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:17.558 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.587 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:17.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:17.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:17.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[29779]: pgmap v803: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 26896/136365 objects misplaced (19.724%); 98 MiB/s, 1.57k objects/s recovering 2024-03-04T16:20:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[29779]: osdmap e374: 8 total, 8 up, 6 in 2024-03-04T16:20:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[29779]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-04T16:20:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[38113]: pgmap v803: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 26896/136365 objects misplaced (19.724%); 98 MiB/s, 1.57k objects/s recovering 2024-03-04T16:20:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[38113]: osdmap e374: 8 total, 8 up, 6 in 2024-03-04T16:20:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:17 smithi038 ceph-mon[38113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-04T16:20:17.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:17 smithi176 ceph-mon[33619]: pgmap v803: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 26896/136365 objects misplaced (19.724%); 98 MiB/s, 1.57k objects/s recovering 2024-03-04T16:20:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:17 smithi176 ceph-mon[33619]: osdmap e374: 8 total, 8 up, 6 in 2024-03-04T16:20:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:17 smithi176 ceph-mon[33619]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-04T16:20:17.867 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:17.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:17.969 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:18.026 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:18.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:18.154 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:18.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:18.299 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:18.400 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:18.426 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:18.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:18.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:18 smithi038 ceph-mon[29779]: pgmap v805: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 26896/136365 objects misplaced (19.724%); 47 MiB/s, 749 objects/s recovering 2024-03-04T16:20:18.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:18 smithi038 ceph-mon[38113]: pgmap v805: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 26896/136365 objects misplaced (19.724%); 47 MiB/s, 749 objects/s recovering 2024-03-04T16:20:18.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.573 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:18.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:18.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:18.712 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:18.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:18.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:18.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:18 smithi176 ceph-mon[33619]: pgmap v805: 158 pgs: 1 peering, 2 active+remapped+backfill_wait, 139 active+clean, 8 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 475 op/s; 26896/136365 objects misplaced (19.724%); 47 MiB/s, 749 objects/s recovering 2024-03-04T16:20:18.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:18.984 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:18.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:19.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:19.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:19.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:19.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:19.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:19.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:19.356 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:19.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:20:19.385 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:19.487 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:19.516 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:19.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:19.617 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.641 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:19.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:19.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:19.782 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:19.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:19.913 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:19.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:20.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:20.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:20.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:20.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.177 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:20.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:20.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:20.302 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:20.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:20.440 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:20.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:20.585 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:20.686 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[29779]: osdmap e375: 8 total, 8 up, 6 in 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[29779]: pgmap v807: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1004 B/s rd, 0 B/s wr, 641 op/s; 20537/126570 objects misplaced (16.226%); 56 MiB/s, 901 objects/s recovering 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[38113]: osdmap e375: 8 total, 8 up, 6 in 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[38113]: pgmap v807: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1004 B/s rd, 0 B/s wr, 641 op/s; 20537/126570 objects misplaced (16.226%); 56 MiB/s, 901 objects/s recovering 2024-03-04T16:20:20.712 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:20 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.730 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:20.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:20.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:20 smithi176 ceph-mon[33619]: osdmap e375: 8 total, 8 up, 6 in 2024-03-04T16:20:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:20 smithi176 ceph-mon[33619]: pgmap v807: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1004 B/s rd, 0 B/s wr, 641 op/s; 20537/126570 objects misplaced (16.226%); 56 MiB/s, 901 objects/s recovering 2024-03-04T16:20:20.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:20 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:20.858 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:20.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:20.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:21.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:21.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:21.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:21.253 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:21.283 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:21.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:21.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:21.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:21.483 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:21 smithi038 ceph-mon[38113]: osdmap e376: 8 total, 8 up, 6 in 2024-03-04T16:20:21.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:21 smithi038 ceph-mon[29779]: osdmap e376: 8 total, 8 up, 6 in 2024-03-04T16:20:21.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:21.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:21.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:21.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:21.721 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:20:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:20:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:20:21.721 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:20:21.721 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:20:21.723 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:20:21.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:21.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:21.763 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:20:21.805 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:20:21.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:21.846 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:20:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:21 smithi176 ceph-mon[33619]: osdmap e376: 8 total, 8 up, 6 in 2024-03-04T16:20:21.869 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:21.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:21.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:21.883 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:20:21.922 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:20:21.961 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:20:21.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:21.998 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:20:21.999 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:20:22.001 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:20:22.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:22.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:22.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:22.268 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:22.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:22.394 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:22.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:22.529 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:22.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[29779]: osdmap e377: 8 total, 8 up, 6 in 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/216019799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[29779]: pgmap v810: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 619 op/s; 20537/126570 objects misplaced (16.226%); 54 MiB/s, 870 objects/s recovering 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[38113]: osdmap e377: 8 total, 8 up, 6 in 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/216019799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:22.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:22 smithi038 ceph-mon[38113]: pgmap v810: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 619 op/s; 20537/126570 objects misplaced (16.226%); 54 MiB/s, 870 objects/s recovering 2024-03-04T16:20:22.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:22.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:22.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:22 smithi176 ceph-mon[33619]: osdmap e377: 8 total, 8 up, 6 in 2024-03-04T16:20:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/216019799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:22 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:22 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:22 smithi176 ceph-mon[33619]: pgmap v810: 158 pgs: 2 active+remapped+backfill_wait, 141 active+clean, 7 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 619 op/s; 20537/126570 objects misplaced (16.226%); 54 MiB/s, 870 objects/s recovering 2024-03-04T16:20:22.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:22.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:22.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:22.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:23.012 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:20:23.012 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:20:23.012 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-04T16:20:23.012 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-04T16:20:23.012 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-04T16:20:23.013 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-04T16:20:23.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:23.084 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:23.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:23.210 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:23.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:23.344 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:23.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:23.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:23.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:23.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:23 smithi176 ceph-mon[33619]: osdmap e378: 8 total, 8 up, 6 in 2024-03-04T16:20:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:23.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:23.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:23.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:23.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:23.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[29779]: osdmap e378: 8 total, 8 up, 6 in 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[38113]: osdmap e378: 8 total, 8 up, 6 in 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:23.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:23.839 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:23.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:23.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:23.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:23.996 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:23.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:24.097 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:24.097 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:24.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:24.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:24.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:24.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:24.376 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:24.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:24.507 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:24 smithi176 ceph-mon[33619]: osdmap e379: 8 total, 8 up, 6 in 2024-03-04T16:20:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:24 smithi176 ceph-mon[33619]: pgmap v813: 158 pgs: 2 active+remapped+backfill_wait, 142 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 18418/122853 objects misplaced (14.992%); 28 MiB/s, 444 objects/s recovering 2024-03-04T16:20:24.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:24 smithi176 ceph-mon[33619]: osdmap e380: 8 total, 8 up, 6 in 2024-03-04T16:20:24.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:24.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[29779]: osdmap e379: 8 total, 8 up, 6 in 2024-03-04T16:20:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[29779]: pgmap v813: 158 pgs: 2 active+remapped+backfill_wait, 142 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 18418/122853 objects misplaced (14.992%); 28 MiB/s, 444 objects/s recovering 2024-03-04T16:20:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[29779]: osdmap e380: 8 total, 8 up, 6 in 2024-03-04T16:20:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[38113]: osdmap e379: 8 total, 8 up, 6 in 2024-03-04T16:20:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[38113]: pgmap v813: 158 pgs: 2 active+remapped+backfill_wait, 142 active+clean, 6 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 310 op/s; 18418/122853 objects misplaced (14.992%); 28 MiB/s, 444 objects/s recovering 2024-03-04T16:20:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:24 smithi038 ceph-mon[38113]: osdmap e380: 8 total, 8 up, 6 in 2024-03-04T16:20:24.745 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:24.771 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:24.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:24.896 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:24.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:24.997 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:25.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:25.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:25.150 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:25.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:25.276 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:25.378 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:25.413 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:25.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:25.557 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:25.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:25.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:25.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:25.816 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:25.918 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:25.942 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:25.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:25.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:20:26.044 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:26.066 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:26.168 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:26.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:26.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:26.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:26.450 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:26.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:26 smithi038 ceph-mon[29779]: osdmap e381: 8 total, 8 up, 6 in 2024-03-04T16:20:26.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:26 smithi038 ceph-mon[38113]: osdmap e381: 8 total, 8 up, 6 in 2024-03-04T16:20:26.520 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:26 smithi176 ceph-mon[33619]: osdmap e381: 8 total, 8 up, 6 in 2024-03-04T16:20:26.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:26.574 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:26.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:26.704 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:26.739 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:20:26.739 INFO:teuthology.orchestra.run.smithi176.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-04T16:20:26.739 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:20:26.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:26.839 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:26.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:26.964 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:26.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:27.065 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:27.088 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:27.190 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:27.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:27.315 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:27.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:27.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:27.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:27.579 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:27.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:27.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:27.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:27 smithi038 ceph-mon[29779]: pgmap v816: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 14097/117987 objects misplaced (11.948%); 83 MiB/s, 1.32k objects/s recovering 2024-03-04T16:20:27.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:27 smithi038 ceph-mon[29779]: osdmap e382: 8 total, 8 up, 6 in 2024-03-04T16:20:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:27 smithi038 ceph-mon[38113]: pgmap v816: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 14097/117987 objects misplaced (11.948%); 83 MiB/s, 1.32k objects/s recovering 2024-03-04T16:20:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:27 smithi038 ceph-mon[38113]: osdmap e382: 8 total, 8 up, 6 in 2024-03-04T16:20:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:27.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:27.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:27 smithi176 ceph-mon[33619]: pgmap v816: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 14097/117987 objects misplaced (11.948%); 83 MiB/s, 1.32k objects/s recovering 2024-03-04T16:20:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:27 smithi176 ceph-mon[33619]: osdmap e382: 8 total, 8 up, 6 in 2024-03-04T16:20:27.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:27.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:27.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:28.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:28.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:28.152 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:28.254 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:28.297 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:28.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:28.428 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:28.529 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:28.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:28.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:28.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:28 smithi038 ceph-mon[29779]: osdmap e383: 8 total, 8 up, 6 in 2024-03-04T16:20:28.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:28 smithi038 ceph-mon[29779]: pgmap v819: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 405 op/s; 14097/117987 objects misplaced (11.948%); 55 MiB/s, 878 objects/s recovering 2024-03-04T16:20:28.587 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:28 smithi038 ceph-mon[38113]: osdmap e383: 8 total, 8 up, 6 in 2024-03-04T16:20:28.588 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:28 smithi038 ceph-mon[38113]: pgmap v819: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 405 op/s; 14097/117987 objects misplaced (11.948%); 55 MiB/s, 878 objects/s recovering 2024-03-04T16:20:28.607 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:28 smithi176 ceph-mon[33619]: osdmap e383: 8 total, 8 up, 6 in 2024-03-04T16:20:28.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:28 smithi176 ceph-mon[33619]: pgmap v819: 158 pgs: 2 active+clean+remapped, 144 active+clean, 4 active+remapped+backfilling, 8 active+remapped+backfill_toofull; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 0 B/s wr, 405 op/s; 14097/117987 objects misplaced (11.948%); 55 MiB/s, 878 objects/s recovering 2024-03-04T16:20:28.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:28.709 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:28.811 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:28.857 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:28.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:28.989 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:28.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:29.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:29.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:29.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:29.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:29.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.433 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:29.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:29.485 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:29.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:29.562 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:29.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.715 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:29.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:29.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:29.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:29.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:29.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:20:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:29.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:30.087 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:30.113 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:30.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:30.267 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:30.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:30.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:30.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:30.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:30.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:30.675 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:30.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:30.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:30.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:30.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:30.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:31.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:31.084 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:31.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:31.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:31.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:31.352 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:31.454 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:31.480 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:31.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:31.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:31.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:31.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:31 smithi038 ceph-mon[29779]: pgmap v820: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 983 B/s rd, 0 B/s wr, 507 op/s; 11302/110076 objects misplaced (10.267%); 24 MiB/s, 386 objects/s recovering 2024-03-04T16:20:31.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:31 smithi038 ceph-mon[38113]: pgmap v820: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 983 B/s rd, 0 B/s wr, 507 op/s; 11302/110076 objects misplaced (10.267%); 24 MiB/s, 386 objects/s recovering 2024-03-04T16:20:31.752 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:31.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:31 smithi176 ceph-mon[33619]: pgmap v820: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 983 B/s rd, 0 B/s wr, 507 op/s; 11302/110076 objects misplaced (10.267%); 24 MiB/s, 386 objects/s recovering 2024-03-04T16:20:31.895 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:31.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:31.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:32.039 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:32.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:32.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:32.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:32.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:32.426 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:32.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:32.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:32.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:20:32.604 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:20:32.604 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-04T16:20:32.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:20:32.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:32.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:32.806 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:32.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:32.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:32.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:33.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:33.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:33.248 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:33.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:33.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:33.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:33.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:33 smithi038 ceph-mon[29779]: pgmap v821: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 812 B/s rd, 0 B/s wr, 419 op/s; 11302/110076 objects misplaced (10.267%); 20 MiB/s, 319 objects/s recovering 2024-03-04T16:20:33.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:33 smithi038 ceph-mon[38113]: pgmap v821: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 812 B/s rd, 0 B/s wr, 419 op/s; 11302/110076 objects misplaced (10.267%); 20 MiB/s, 319 objects/s recovering 2024-03-04T16:20:33.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:33.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:33.607 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:33 smithi176 ceph-mon[33619]: pgmap v821: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 812 B/s rd, 0 B/s wr, 419 op/s; 11302/110076 objects misplaced (10.267%); 20 MiB/s, 319 objects/s recovering 2024-03-04T16:20:33.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:33.709 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:33.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:33.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:33.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:33.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:34.034 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:34.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:34.162 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:34.263 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:34.312 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:34.338 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:34.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:20:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.479 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:34.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:34.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:20:34.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:34.494 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:20:34.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:34.634 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:34.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:34.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:20:34.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:34.932 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:34.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:35.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:35.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:35.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:35.217 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:35.319 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:35.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:35.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:35.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: pgmap v822: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 10987/104448 objects misplaced (10.519%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:35.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:35.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:20:35.406 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: osdmap e384: 8 total, 8 up, 6 in 2024-03-04T16:20:35.463 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:35.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:35.609 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:35.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:35.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: pgmap v822: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 10987/104448 objects misplaced (10.519%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: osdmap e384: 8 total, 8 up, 6 in 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 10.9 scrub starts 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 10.9 scrub ok 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 10.7 scrub starts 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: 10.7 scrub ok 2024-03-04T16:20:35.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[29779]: osdmap e385: 8 total, 8 up, 6 in 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 10.9 scrub starts 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 10.9 scrub ok 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 10.7 scrub starts 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: 10.7 scrub ok 2024-03-04T16:20:35.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:35 smithi038 ceph-mon[38113]: osdmap e385: 8 total, 8 up, 6 in 2024-03-04T16:20:35.740 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:35.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: pgmap v822: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 2.1 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 10987/104448 objects misplaced (10.519%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2189321501' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: osdmap e384: 8 total, 8 up, 6 in 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 10.9 scrub starts 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 10.9 scrub ok 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 10.7 scrub starts 2024-03-04T16:20:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:20:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:20:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: 10.7 scrub ok 2024-03-04T16:20:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:35 smithi176 ceph-mon[33619]: osdmap e385: 8 total, 8 up, 6 in 2024-03-04T16:20:35.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:35.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:35.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:36.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:36.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:36.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:36.213 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:36.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.364 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:36.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:36.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:36.508 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:36.610 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.638 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:36.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:36.706 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:20:36.706 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:20:36.706 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:20:36.706 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: pgmap v825: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 9714/94803 objects misplaced (10.247%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: pgmap v825: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 9714/94803 objects misplaced (10.247%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:20:36.707 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:36 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:20:36.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:36.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: pgmap v825: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 966 op/s; 9714/94803 objects misplaced (10.247%); 16 MiB/s, 251 objects/s recovering 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:20:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:36 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:20:36.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:36.921 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:36.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:37.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.049 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:37.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:37.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:37.292 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:37.320 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:37.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:37.451 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:37.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:37.597 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:37.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:20:37.699 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[29779]: 10.10 scrub starts 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[29779]: 10.10 scrub ok 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[29779]: 10.15 scrub starts 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[29779]: 10.15 scrub ok 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[38113]: 10.10 scrub starts 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[38113]: 10.10 scrub ok 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[38113]: 10.15 scrub starts 2024-03-04T16:20:37.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:37 smithi038 ceph-mon[38113]: 10.15 scrub ok 2024-03-04T16:20:37.725 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:37.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:37.851 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:37.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:37 smithi176 ceph-mon[33619]: 10.10 scrub starts 2024-03-04T16:20:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:37 smithi176 ceph-mon[33619]: 10.10 scrub ok 2024-03-04T16:20:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:37 smithi176 ceph-mon[33619]: 10.15 scrub starts 2024-03-04T16:20:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:37 smithi176 ceph-mon[33619]: 10.15 scrub ok 2024-03-04T16:20:37.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:37.964 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:38.009 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:38.111 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:38.134 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:38.236 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.276 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:38.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:38.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:38.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:38.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:38.532 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:38.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:38.676 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:38.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: 10.14 scrub starts 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: 10.14 scrub ok 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: 10.1 scrub starts 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: 10.1 scrub ok 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: pgmap v826: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 9714/94803 objects misplaced (10.247%) 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: 10.14 scrub starts 2024-03-04T16:20:38.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: 10.14 scrub ok 2024-03-04T16:20:38.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: 10.1 scrub starts 2024-03-04T16:20:38.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: 10.1 scrub ok 2024-03-04T16:20:38.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: pgmap v826: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 9714/94803 objects misplaced (10.247%) 2024-03-04T16:20:38.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:38.778 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.804 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:38.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:38.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:20:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: 10.14 scrub starts 2024-03-04T16:20:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: 10.14 scrub ok 2024-03-04T16:20:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: 10.1 scrub starts 2024-03-04T16:20:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: 10.1 scrub ok 2024-03-04T16:20:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: pgmap v826: 158 pgs: 150 active+clean, 8 active+remapped+backfill_toofull; 1.9 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s; 9714/94803 objects misplaced (10.247%) 2024-03-04T16:20:38.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:38.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:38.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:38.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:39.051 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:39.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:39.135 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:39.180 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:39.210 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:39.312 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:39.356 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:39.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:39.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:39.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:39 smithi176 ceph-mon[33619]: 10.13 scrub starts 2024-03-04T16:20:39.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:39 smithi176 ceph-mon[33619]: 10.13 scrub ok 2024-03-04T16:20:39.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:39 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:20:39.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:39 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:20:39.539 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:39 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:39.587 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:39.614 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:39.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:39.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[29779]: 10.13 scrub starts 2024-03-04T16:20:39.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[29779]: 10.13 scrub ok 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[38113]: 10.13 scrub starts 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[38113]: 10.13 scrub ok 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:20:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:39 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:39.761 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:39.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:39.891 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:39.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:39.993 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:40.042 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:40.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:40.188 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:40.290 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:40.316 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:40.387 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:40.418 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:40.446 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:40.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:40.592 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:40.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:40 smithi176 ceph-mon[33619]: 10.19 scrub starts 2024-03-04T16:20:40.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:40 smithi176 ceph-mon[33619]: 10.19 scrub ok 2024-03-04T16:20:40.593 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:40 smithi176 ceph-mon[33619]: pgmap v827: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:40.694 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:40.732 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:40.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:40.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[29779]: 10.19 scrub starts 2024-03-04T16:20:40.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[29779]: 10.19 scrub ok 2024-03-04T16:20:40.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[29779]: pgmap v827: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:40.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[38113]: 10.19 scrub starts 2024-03-04T16:20:40.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[38113]: 10.19 scrub ok 2024-03-04T16:20:40.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:40 smithi038 ceph-mon[38113]: pgmap v827: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:40.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:40.880 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:40.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:40.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.024 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:41.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:41.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:41.155 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:41.221 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:20:41.221 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:20:41.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:41.300 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:41.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:41.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:41.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:41.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:41 smithi176 ceph-mon[33619]: 10.0 scrub starts 2024-03-04T16:20:41.552 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:41 smithi176 ceph-mon[33619]: 10.0 scrub ok 2024-03-04T16:20:41.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:41 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:41.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:41 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:20:41.553 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:41 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:20:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.573 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:41.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:41.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[29779]: 10.0 scrub starts 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[29779]: 10.0 scrub ok 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:20:41.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[38113]: 10.0 scrub starts 2024-03-04T16:20:41.706 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[38113]: 10.0 scrub ok 2024-03-04T16:20:41.706 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:41.706 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:20:41.706 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:41 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:20:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:41.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:41.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:41.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:41.895 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:20:41.944 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:41.974 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:41.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:42.076 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:42.124 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:42.204 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:42.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:42.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:42.381 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:42.408 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:42.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:42.537 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:42.606 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:42 smithi176 ceph-mon[33619]: 10.2 scrub starts 2024-03-04T16:20:42.606 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:42 smithi176 ceph-mon[33619]: 10.2 scrub ok 2024-03-04T16:20:42.606 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:42 smithi176 ceph-mon[33619]: osdmap e386: 8 total, 8 up, 6 in 2024-03-04T16:20:42.606 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:42 smithi176 ceph-mon[33619]: pgmap v829: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 704 B/s rd, 0 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:42.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.691 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:42.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[29779]: 10.2 scrub starts 2024-03-04T16:20:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[29779]: 10.2 scrub ok 2024-03-04T16:20:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[29779]: osdmap e386: 8 total, 8 up, 6 in 2024-03-04T16:20:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[29779]: pgmap v829: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 704 B/s rd, 0 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[38113]: 10.2 scrub starts 2024-03-04T16:20:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[38113]: 10.2 scrub ok 2024-03-04T16:20:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[38113]: osdmap e386: 8 total, 8 up, 6 in 2024-03-04T16:20:42.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:42 smithi038 ceph-mon[38113]: pgmap v829: 158 pgs: 4 active+remapped+backfilling, 150 active+clean, 4 active+remapped+backfill_toofull; 1.6 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 704 B/s rd, 0 op/s; 7907/79758 objects misplaced (9.914%) 2024-03-04T16:20:42.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.834 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:42.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:42.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:42.977 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:42.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:43.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:43.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:43.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:43.249 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:43.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:43.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:43.445 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:43.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:43.536 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:43.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:43 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:43.608 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:43.638 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.682 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:43.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:43.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:43 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:43.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:43 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:43.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:43.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:43.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:43.939 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:43.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:43.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:44.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:44.085 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:20:44.126 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:20:44.126 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:20:44.170 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:20:44.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:44.204 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:20:44.204 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:20:44.204 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 386, 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:20:41.468368+0000", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:20:06.398407+0000", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5804032, 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "nodeep-scrub", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:20:44.205 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 10, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:20:44.206 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:20:44.207 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "279", 2024-03-04T16:20:44.208 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.209 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.210 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:20:44.211 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.213 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.214 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:20:44.215 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.216 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.217 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6700000762939453, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6700000762939453, 2024-03-04T16:20:44.218 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:20:44.220 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.221 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.222 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.223 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.225 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:20:44.226 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:20:44.227 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:20:44.228 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.229 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.231 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.5099999904632568, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.5099999904632568, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.233 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.234 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:20:44.237 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.238 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.239 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.240 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.241 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.242 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.243 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5, 2024-03-04T16:20:44.244 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 10, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_3", 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:16:52.586520+0000", 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:20:44.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 26, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 26, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 26, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 26, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 26, 2024-03-04T16:20:44.246 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "322", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "320", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:20:44.247 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:20:44.248 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:20:44.249 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.8400000333786011, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.8400000333786011, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.75, 2024-03-04T16:20:44.250 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.3799999952316284, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.3799999952316284, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.9722900390625, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.251 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 378, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:20:44.252 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.253 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:20:44.254 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.255 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.256 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:44.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.257 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 382, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:20:44.258 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.259 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:20:44.260 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.261 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 380, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.262 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.263 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.264 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.265 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 373, 2024-03-04T16:20:44.266 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.268 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:20:44.269 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.270 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 358, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.271 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.272 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.273 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.274 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 365, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.275 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.278 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 190, 2024-03-04T16:20:44.279 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.280 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:20:44.281 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.282 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:20:44.283 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 379, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:20:44.284 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:20:44.285 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:20:44.286 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.287 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:19:45.810062+0000", 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:20:44.288 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:20:08.584725+0000", 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.289 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:19:58.894019+0000", 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:20:11.779086+0000", 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:20:40.568983+0000", 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:20:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:20:13.863340+0000", 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:20:21.669727+0000", 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:19:43.449427+0000", 2024-03-04T16:20:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [], 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.8", 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 3 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.13", 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [ 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.3", 2024-03-04T16:20:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.6", 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.a", 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: 3 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.d", 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:20:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.12", 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.16", 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.17", 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 3, 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 1, 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: 4 2024-03-04T16:20:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "10.18", 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: 7 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:20:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:20:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:20:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:20:44.303 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:20:44.304 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:20:44.323 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:20:44.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:44.368 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:20:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:44.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:44.410 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:20:44.449 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:20:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:20:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:20:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:20:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:20:44.460 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:20:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:20:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:20:44.461 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:20:44.507 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:44.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:44.651 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[38113]: pgmap v830: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.5 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7612/74391 objects misplaced (10.232%) 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2519093227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1911444738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[29779]: pgmap v830: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.5 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7612/74391 objects misplaced (10.232%) 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2519093227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:44.691 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1911444738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:20:44.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.693 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:44.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:44.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:44.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:44.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:44 smithi176 ceph-mon[33619]: pgmap v830: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.5 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 7612/74391 objects misplaced (10.232%) 2024-03-04T16:20:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2519093227' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1911444738' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:20:44.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:44.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:44.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:45.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:45.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:45.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:20:45.202 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:45.231 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:45.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:45.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:45.435 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:45.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:45.520 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:45.622 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:45.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:45.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:45.780 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:45.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:20:45.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:20:45.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[29779]: osdmap e387: 8 total, 8 up, 6 in 2024-03-04T16:20:45.786 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:20:45.786 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:20:45.786 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:45 smithi038 ceph-mon[38113]: osdmap e387: 8 total, 8 up, 6 in 2024-03-04T16:20:45.819 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:45 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:20:45.819 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:45 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:20:45.819 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:45 smithi176 ceph-mon[33619]: osdmap e387: 8 total, 8 up, 6 in 2024-03-04T16:20:45.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:45.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:45.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:45.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:46.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:46.052 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:46.154 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:46.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:46.344 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:46.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:46.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:46.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:46.629 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:46.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:46.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:20:46.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:20:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[29779]: pgmap v832: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[29779]: osdmap e388: 8 total, 8 up, 6 in 2024-03-04T16:20:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:20:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:20:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[38113]: pgmap v832: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:46 smithi038 ceph-mon[38113]: osdmap e388: 8 total, 8 up, 6 in 2024-03-04T16:20:46.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:46.791 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:46 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:20:46.791 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:46 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:20:46.791 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:46 smithi176 ceph-mon[33619]: pgmap v832: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:46.791 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:46 smithi176 ceph-mon[33619]: osdmap e388: 8 total, 8 up, 6 in 2024-03-04T16:20:46.861 INFO:tasks.thrashosds.thrasher:j is {'epoch': 386, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:20:41.468368+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:20:06.398407+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-04T16:16:52.586520+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '322', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '320', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.840000033378601, 'score_stable': 1.840000033378601, 'optimal_score': 0.75, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 0.9722900390625, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 378, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 373, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 358, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 190, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 379, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:19:45.810062+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:20:08.584725+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:19:58.894019+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:20:11.779086+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:20:40.568983+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:20:13.863340+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:20:21.669727+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:19:43.449427+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '5.8', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 0, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.3', 'osds': [3, 5, 4]}, {'pgid': '10.6', 'osds': [5, 0, 4]}, {'pgid': '10.a', 'osds': [0, 5, 3]}, {'pgid': '10.d', 'osds': [0, 3, 4]}, {'pgid': '10.12', 'osds': [0, 1, 4]}, {'pgid': '10.16', 'osds': [5, 1, 4]}, {'pgid': '10.17', 'osds': [3, 1, 4]}, {'pgid': '10.18', 'osds': [0, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:20:46.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:20:46.875 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:46.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:46.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:47.047 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:47.075 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:47.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:47.202 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:153942 10.17 10:e9e32678:::benchmark_data_smithi038_155391_object17985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-04T16:20:47.212 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:18:50.415822+0000", 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.793568167, 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94555652199999995, 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 153942 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.213 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415822+0000", 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415822+0000", 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415823+0000", 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5069999999999999e-06 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.214 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415829+0000", 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8669999999999999e-06 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415843+0000", 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3492000000000001e-05 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.215 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.415851+0000", 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7589999999999997e-06 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.416205+0000", 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035486900000000001 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.216 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.416214+0000", 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3829999999999992e-06 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.355315+0000", 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93910126000000005 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.217 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.355458+0000", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014341899999999999 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.355762+0000", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00030333399999999999 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.357268+0000", 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001506428 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.218 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361255+0000", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039869880000000003 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361313+0000", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7559000000000002e-05 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361342+0000", 2024-03-04T16:20:47.219 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8949e-05 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361378+0000", 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6708000000000002e-05 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:153945 10.17 10:ebef4932:::benchmark_data_smithi038_155391_object17988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:18:50.417340+0000", 2024-03-04T16:20:47.220 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.79204990700001, 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95205815999999999, 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 153945 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417340+0000", 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.221 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417340+0000", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417341+0000", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0860000000000001e-06 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417346+0000", 2024-03-04T16:20:47.222 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3089999999999997e-06 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417355+0000", 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6320000000000003e-06 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417361+0000", 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7470000000000002e-06 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.223 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417407+0000", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5881999999999999e-05 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.417417+0000", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.03e-05 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.357305+0000", 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93988823399999999 2024-03-04T16:20:47.224 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.357392+0000", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6598e-05 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.357557+0000", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000165437 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361207+0000", 2024-03-04T16:20:47.225 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003649195 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369247+0000", 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080400569999999998 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369323+0000", 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6297000000000005e-05 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.226 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369355+0000", 2024-03-04T16:20:47.227 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1835999999999998e-05 2024-03-04T16:20:47.227 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.227 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.227 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369398+0000", 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3550000000000001e-05 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:153959 10.17 10:eea667b8:::benchmark_data_smithi038_155391_object18002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-04T16:20:47.228 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:18:50.429129+0000", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.780260673, 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94911306299999998, 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 153959 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429129+0000", 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.229 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429129+0000", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429131+0000", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.731e-06 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429137+0000", 2024-03-04T16:20:47.230 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3870000000000003e-06 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429146+0000", 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5650000000000007e-06 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429153+0000", 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7560000000000002e-06 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.231 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429264+0000", 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011090399999999999 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.429272+0000", 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2900000000000002e-06 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.358317+0000", 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92904505100000001 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.232 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.358428+0000", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000110711 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.358613+0000", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018555399999999999 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361387+0000", 2024-03-04T16:20:47.233 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027738509999999999 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369298+0000", 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.007910828 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378126+0000", 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088275030000000004 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.234 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378186+0000", 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0785000000000002e-05 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378242+0000", 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6147000000000003e-05 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.235 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:154020 10.17 10:e913f46b:::benchmark_data_smithi038_155391_object18063:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:18:50.468350+0000", 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.74104030399999, 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.91005822300000005, 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 154020 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.236 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468350+0000", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468350+0000", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468351+0000", 2024-03-04T16:20:47.237 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.689e-06 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468367+0000", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5778999999999999e-05 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468381+0000", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4088e-05 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468389+0000", 2024-03-04T16:20:47.238 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1110000000000008e-06 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468454+0000", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4849000000000002e-05 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.468462+0000", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2320000000000001e-06 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.359326+0000", 2024-03-04T16:20:47.239 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.89086388299999997 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.359434+0000", 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107272 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.359590+0000", 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015604700000000001 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.240 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.361391+0000", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018011209999999999 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369454+0000", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0080634109999999995 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378316+0000", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0088619240000000002 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.241 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378373+0000", 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6441999999999999e-05 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378408+0000", 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5374999999999999e-05 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:154070 10.17 10:edaa9568:::benchmark_data_smithi038_155391_object18113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:18:50.502428+0000", 2024-03-04T16:20:47.242 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 116.706962046, 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87607227700000001, 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 154070 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502428+0000", 2024-03-04T16:20:47.243 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502428+0000", 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502458+0000", 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9791e-05 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.244 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502490+0000", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1826000000000003e-05 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502499+0000", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6670000000000008e-06 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502507+0000", 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2960000000000001e-06 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.245 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502587+0000", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0708e-05 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:50.502597+0000", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0169e-05 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.360340+0000", 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.857742581 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.246 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.360426+0000", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6420000000000003e-05 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,3", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.360564+0000", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013727499999999999 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.363276+0000", 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002712098 2024-03-04T16:20:47.247 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.369490+0000", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0062140269999999996 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378447+0000", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0089571860000000007 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378478+0000", 2024-03-04T16:20:47.248 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0562999999999999e-05 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:18:51.378500+0000", 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2670000000000001e-05 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:179280 10.10 10:0e81d4a7:::benchmark_data_smithi038_155391_object43323:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:16.376021+0000", 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 90.833368590999996, 2024-03-04T16:20:47.249 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.012206266, 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179280 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376021+0000", 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.250 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376021+0000", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376023+0000", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.691e-06 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376028+0000", 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3140000000000003e-06 2024-03-04T16:20:47.251 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376039+0000", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0424999999999999e-05 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376047+0000", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3860000000000007e-06 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376110+0000", 2024-03-04T16:20:47.252 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2242000000000005e-05 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.376119+0000", 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8900000000000002e-06 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.382899+0000", 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0067797060000001 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.253 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.383009+0000", 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010946300000000001 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.383350+0000", 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034125900000000001 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388036+0000", 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046863929999999996 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.254 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388084+0000", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7335e-05 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388134+0000", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0702999999999999e-05 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388185+0000", 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0308e-05 2024-03-04T16:20:47.255 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388228+0000", 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3151000000000001e-05 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:179296 10.10 10:0e54b194:::benchmark_data_smithi038_155391_object43339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:16.395578+0000", 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 90.813812170999995, 2024-03-04T16:20:47.256 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99436624500000004, 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179296 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395578+0000", 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.257 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395578+0000", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395580+0000", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7e-06 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395585+0000", 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5690000000000004e-06 2024-03-04T16:20:47.258 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395598+0000", 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2627e-05 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395606+0000", 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3930000000000008e-06 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.259 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395656+0000", 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9432999999999999e-05 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.395661+0000", 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7740000000000001e-06 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.385080+0000", 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.989418982 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.260 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.385183+0000", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102404 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.385393+0000", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021026899999999999 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389750+0000", 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043572749999999999 2024-03-04T16:20:47.261 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389791+0000", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0509000000000001e-05 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389849+0000", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7734e-05 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389905+0000", 2024-03-04T16:20:47.262 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6118e-05 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389944+0000", 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9458000000000001e-05 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:179306 10.10 10:0f2e7624:::benchmark_data_smithi038_155391_object43349:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:16.403241+0000", 2024-03-04T16:20:47.263 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 90.806148690000001, 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99048027699999996, 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179306 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403241+0000", 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.264 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403241+0000", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403243+0000", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1599999999999999e-06 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403247+0000", 2024-03-04T16:20:47.265 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9450000000000001e-06 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403260+0000", 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2587e-05 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403268+0000", 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1769999999999998e-06 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.266 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403330+0000", 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2261999999999994e-05 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.403339+0000", 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6519999999999995e-06 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.386443+0000", 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98310408100000002 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.267 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.386516+0000", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2679e-05 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.386726+0000", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021026399999999999 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389806+0000", 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003080251 2024-03-04T16:20:47.268 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393578+0000", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003771286 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393615+0000", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7280000000000002e-05 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393663+0000", 2024-03-04T16:20:47.269 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8219e-05 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393722+0000", 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8434000000000003e-05 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:179320 10.10 10:08193174:::benchmark_data_smithi038_155391_object43363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:16.416109+0000", 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 90.793280577999994, 2024-03-04T16:20:47.270 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97957087499999995, 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179320 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416109+0000", 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.271 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416109+0000", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416111+0000", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4300000000000001e-06 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416116+0000", 2024-03-04T16:20:47.272 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2039999999999996e-06 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416127+0000", 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0458e-05 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416134+0000", 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3069999999999997e-06 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.273 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416183+0000", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9234000000000003e-05 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.416189+0000", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6489999999999996e-06 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388253+0000", 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97206392200000002 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.274 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388336+0000", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3474999999999996e-05 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.388484+0000", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014819900000000001 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393762+0000", 2024-03-04T16:20:47.275 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052780209999999999 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393798+0000", 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5348000000000001e-05 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.395581+0000", 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001783083 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.276 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.395632+0000", 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1597000000000001e-05 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.395680+0000", 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7948000000000002e-05 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:179377 10.10 10:0edb8d49:::benchmark_data_smithi038_155391_object43420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-04T16:20:47.277 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:16.465444+0000", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 90.743946085000005, 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.931966497, 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 179377 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465444+0000", 2024-03-04T16:20:47.278 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465444+0000", 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465445+0000", 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4589999999999999e-06 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.279 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465450+0000", 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7709999999999997e-06 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465503+0000", 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2377999999999999e-05 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465510+0000", 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.515e-06 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.280 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465563+0000", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3155000000000002e-05 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.465573+0000", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3710000000000006e-06 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.389977+0000", 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92440461600000001 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.281 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.390089+0000", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011156500000000001 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.390351+0000", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026205299999999999 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.393834+0000", 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003482822 2024-03-04T16:20:47.282 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.397266+0000", 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034319279999999999 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.283 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.284 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.397347+0000", 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0963000000000004e-05 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.397381+0000", 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4827999999999999e-05 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.285 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:17.397410+0000", 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9073000000000001e-05 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209221 10.e 10:7e273b3e:::benchmark_data_smithi038_155391_object73264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.647169+0000", 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.562220697000001, 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95796511799999995, 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.286 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209221 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647169+0000", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647169+0000", 2024-03-04T16:20:47.287 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647171+0000", 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.221e-06 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647175+0000", 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7899999999999999e-06 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.288 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647184+0000", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5979999999999997e-06 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647191+0000", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1919999999999999e-06 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647245+0000", 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4197999999999998e-05 2024-03-04T16:20:47.289 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.647254+0000", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.653e-06 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.597779+0000", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95052479099999998 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.597869+0000", 2024-03-04T16:20:47.290 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0099000000000006e-05 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.597947+0000", 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8195000000000006e-05 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.601214+0000", 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032669600000000002 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.291 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.602719+0000", 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015053040000000001 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605059+0000", 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023400560000000001 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605093+0000", 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3417000000000002e-05 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.292 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605134+0000", 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1644000000000003e-05 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209225 10.e 10:71298325:::benchmark_data_smithi038_155391_object73268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.652749+0000", 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.556641042000003, 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95249214900000001, 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.293 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209225 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652749+0000", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652749+0000", 2024-03-04T16:20:47.294 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652751+0000", 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2220000000000001e-06 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652757+0000", 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.75e-06 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.295 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652768+0000", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0593e-05 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652776+0000", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0790000000000001e-06 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652835+0000", 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9058999999999998e-05 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.296 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.652844+0000", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4180000000000003e-06 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.599899+0000", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94705465899999997 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.600005+0000", 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0001059 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.297 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.600092+0000", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6979999999999997e-05 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.602824+0000", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027319390000000001 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.603917+0000", 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001093723 2024-03-04T16:20:47.298 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605190+0000", 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001272277 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605217+0000", 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7619000000000001e-05 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.299 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605241+0000", 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3930999999999998e-05 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209237 10.e 10:7b582d4e:::benchmark_data_smithi038_155391_object73280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.661045+0000", 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.548345220000002, 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94699818999999996, 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.300 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209237 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661045+0000", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.301 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661045+0000", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661046+0000", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1489999999999999e-06 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661051+0000", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7940000000000002e-06 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.302 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661061+0000", 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0213e-05 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661068+0000", 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1609999999999997e-06 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661132+0000", 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3421000000000001e-05 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.303 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.661139+0000", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3640000000000001e-06 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.601271+0000", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94013197999999998 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.601401+0000", 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000130018 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.304 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.601537+0000", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013651399999999999 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.604965+0000", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034271969999999999 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605020+0000", 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4880000000000003e-05 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.305 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.607975+0000", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002955268 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608011+0000", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6270999999999999e-05 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608043+0000", 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.1959999999999999e-05 2024-03-04T16:20:47.306 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209247 10.e 10:7b695c0d:::benchmark_data_smithi038_155391_object73290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.672938+0000", 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.536452443999998, 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.937550934, 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.307 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209247 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672938+0000", 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672938+0000", 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.308 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672939+0000", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8139999999999999e-06 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672945+0000", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7189999999999998e-06 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672958+0000", 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.279e-05 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.309 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.672966+0000", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8269999999999998e-06 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.673015+0000", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9434000000000001e-05 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.673021+0000", 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1589999999999998e-06 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.310 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.602856+0000", 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92983452799999999 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.602951+0000", 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5475000000000003e-05 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.311 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.603016+0000", 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4309999999999999e-05 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605143+0000", 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002127234 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.607900+0000", 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027573480000000002 2024-03-04T16:20:47.312 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.610429+0000", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025291139999999998 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.610462+0000", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.286e-05 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.610489+0000", 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6322e-05 2024-03-04T16:20:47.313 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209254 10.e 10:770d3969:::benchmark_data_smithi038_155391_object73297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.677298+0000", 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.532092022999997, 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93619465999999996, 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.314 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209254 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677298+0000", 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677298+0000", 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.315 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677300+0000", 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.722e-06 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677305+0000", 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6030000000000004e-06 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.316 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677317+0000", 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2092e-05 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677325+0000", 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6669999999999996e-06 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677382+0000", 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7289e-05 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.317 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.677394+0000", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1729000000000001e-05 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.603936+0000", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92654229799999999 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.604007+0000", 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0492000000000002e-05 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.318 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.604070+0000", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2897000000000004e-05 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.606377+0000", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023073479999999999 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.607937+0000", 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001559567 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.319 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.613396+0000", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054590760000000002 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.613448+0000", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2473999999999999e-05 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.613493+0000", 2024-03-04T16:20:47.320 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4406000000000001e-05 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209262 10.e 10:769d131d:::benchmark_data_smithi038_155391_object73305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.683809+0000", 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.52558106, 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93385466399999995, 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.321 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209262 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683809+0000", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683809+0000", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683811+0000", 2024-03-04T16:20:47.322 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.874e-06 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683817+0000", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8100000000000003e-06 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683827+0000", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.825e-06 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683835+0000", 2024-03-04T16:20:47.323 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5490000000000006e-06 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683891+0000", 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6382000000000003e-05 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.683901+0000", 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6129999999999993e-06 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.324 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605277+0000", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92137614700000003 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605363+0000", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5997999999999999e-05 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.605429+0000", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6091000000000001e-05 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.325 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608050+0000", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0026205740000000001 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609207+0000", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0011572189999999999 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617583+0000", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0083755710000000001 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.326 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617620+0000", 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7633999999999999e-05 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617664+0000", 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3377000000000002e-05 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209278 10.e 10:790a2870:::benchmark_data_smithi038_155391_object73321:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.701458+0000", 2024-03-04T16:20:47.327 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.507932545999999, 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.91644417499999997, 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209278 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701458+0000", 2024-03-04T16:20:47.328 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701458+0000", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701459+0000", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5740000000000001e-06 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701464+0000", 2024-03-04T16:20:47.329 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9150000000000004e-06 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701473+0000", 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6179999999999995e-06 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701479+0000", 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5699999999999998e-06 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.330 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701575+0000", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5950000000000001e-05 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.701584+0000", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9199999999999993e-06 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.606420+0000", 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90483643199999997 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.331 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.606525+0000", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010433 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.606638+0000", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113337 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609168+0000", 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025300750000000001 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.332 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609247+0000", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8910000000000002e-05 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617777+0000", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0085295490000000009 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617856+0000", 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9008e-05 2024-03-04T16:20:47.333 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617902+0000", 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5986999999999999e-05 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209300 10.e 10:7843f135:::benchmark_data_smithi038_155391_object73343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.729202+0000", 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.480187655999998, 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.88885331899999998, 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.334 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209300 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729202+0000", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729202+0000", 2024-03-04T16:20:47.335 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729204+0000", 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.455e-06 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729209+0000", 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.336 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729219+0000", 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9140000000000003e-06 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729225+0000", 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9229999999999999e-06 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729276+0000", 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0852e-05 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.337 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.729296+0000", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9857e-05 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608067+0000", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87877119699999995 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608133+0000", 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6024000000000003e-05 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.338 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.608202+0000", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9258999999999995e-05 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.610537+0000", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023350720000000001 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.610547+0000", 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3870000000000001e-06 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.339 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.617991+0000", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074446470000000004 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.618022+0000", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.0882e-05 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.618056+0000", 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3531000000000001e-05 2024-03-04T16:20:47.340 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:209314 10.e 10:7f117b71:::benchmark_data_smithi038_155391_object73357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e358)", 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:50.741593+0000", 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 56.467796730000003, 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.88317285300000004, 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:47.341 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 209314 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741593+0000", 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741593+0000", 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.342 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741595+0000", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.844e-06 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741600+0000", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.327e-06 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741616+0000", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5153e-05 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.343 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741623+0000", 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.6699999999999994e-06 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741678+0000", 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4273e-05 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:50.741687+0000", 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1609999999999992e-06 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.344 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609274+0000", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.86758726600000002 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609343+0000", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9016999999999993e-05 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.609436+0000", 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3209999999999997e-05 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.345 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.613356+0000", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039193650000000002 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.613537+0000", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018183300000000001 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.624676+0000", 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011138301 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.346 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.624725+0000", 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9134000000000001e-05 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:51.624766+0000", 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1298999999999999e-05 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.347 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.35585.0:252755 10.d e387/365 10:bd8d1971:::benchmark_data_smithi038_155391_object116798:head v 387'15716, mlcod=387'15712)", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:46.181566+0000", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 1.027824157, 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049859250000000004, 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181566+0000", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:47.348 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181566+0000", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181566+0000", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4600000000000005e-07 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181571+0000", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0150000000000003e-06 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:47.349 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181578+0000", 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.314e-06 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.181583+0000", 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3000000000000001e-06 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.184279+0000", 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002695888 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.350 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.184314+0000", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5049000000000003e-05 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.186507+0000", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021930180000000001 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:46.186552+0000", 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4795000000000001e-05 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.351 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.352 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:47.352 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:47.352 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:20:47.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:47.377 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:47.479 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:47.508 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:47.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.655 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:47.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:47.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:47.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:47.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:20:47.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:47.923 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:47.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:48.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:48.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:48.165 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.189 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:48.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:48.226 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:48.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:48.314 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:48.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:48.428 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:20:48.448 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:48.550 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:48.573 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:48.674 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:48.698 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:48.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:48.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[38113]: osdmap e389: 8 total, 8 up, 6 in 2024-03-04T16:20:48.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3192814985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[38113]: from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[38113]: pgmap v835: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[38113]: 10.3 scrub starts 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[29779]: osdmap e389: 8 total, 8 up, 6 in 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3192814985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[29779]: from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[29779]: pgmap v835: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:48.719 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:48 smithi038 ceph-mon[29779]: 10.3 scrub starts 2024-03-04T16:20:48.800 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:48.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:48.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:48 smithi176 ceph-mon[33619]: osdmap e389: 8 total, 8 up, 6 in 2024-03-04T16:20:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3192814985' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:48 smithi176 ceph-mon[33619]: from='client.25717 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:20:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:48 smithi176 ceph-mon[33619]: pgmap v835: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean, 3 active+remapped+backfill_toofull; 1.3 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 4960/65640 objects misplaced (7.556%) 2024-03-04T16:20:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:48 smithi176 ceph-mon[33619]: 10.3 scrub starts 2024-03-04T16:20:48.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:48.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:48.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:49.050 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.075 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:49.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:49.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:49.200 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:49.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:20:49.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:49.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:49.428 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:49.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:49.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:49.569 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:49.594 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: 10.3 scrub ok 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: osdmap e390: 8 total, 8 up, 6 in 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: 10.17 scrub starts 2024-03-04T16:20:49.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:49 smithi176 ceph-mon[33619]: 10.17 scrub ok 2024-03-04T16:20:49.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:49.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:49.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:20:49.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: 10.3 scrub ok 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: osdmap e390: 8 total, 8 up, 6 in 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: 10.17 scrub starts 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[29779]: 10.17 scrub ok 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: 10.3 scrub ok 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: from='client.25717 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: osdmap e390: 8 total, 8 up, 6 in 2024-03-04T16:20:49.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: 10.17 scrub starts 2024-03-04T16:20:49.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:49 smithi038 ceph-mon[38113]: 10.17 scrub ok 2024-03-04T16:20:49.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:49.849 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:49.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:49.997 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:49.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:50.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:50.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:50.236 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:50.263 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:50.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:50.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:50.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:50.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:50 smithi176 ceph-mon[33619]: 10.8 deep-scrub starts 2024-03-04T16:20:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:50 smithi176 ceph-mon[33619]: osdmap e391: 8 total, 8 up, 6 in 2024-03-04T16:20:50.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:50 smithi176 ceph-mon[33619]: pgmap v838: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:50.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.652 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:50.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:50.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[29779]: 10.8 deep-scrub starts 2024-03-04T16:20:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[29779]: osdmap e391: 8 total, 8 up, 6 in 2024-03-04T16:20:50.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[29779]: pgmap v838: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[38113]: 10.8 deep-scrub starts 2024-03-04T16:20:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[38113]: osdmap e391: 8 total, 8 up, 6 in 2024-03-04T16:20:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:50 smithi038 ceph-mon[38113]: pgmap v838: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:50.753 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:50.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:50.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:50.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:50.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:50.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:51.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:51.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:51.152 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:51.178 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:51.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:51.306 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:51.408 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:51.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:51.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:51.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:51.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:51.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:20:51.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:20:51.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:20:51.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:20:51.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 10.5 scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 10.16 scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 10.5 scrub ok 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[29779]: 10.16 scrub ok 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 10.5 scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 10.16 scrub starts 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 10.5 scrub ok 2024-03-04T16:20:51.715 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:51 smithi038 ceph-mon[38113]: 10.16 scrub ok 2024-03-04T16:20:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:51.821 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:51.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:20:51.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:20:51.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 10.5 scrub starts 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 10.16 scrub starts 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 10.5 scrub ok 2024-03-04T16:20:51.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:51 smithi176 ceph-mon[33619]: 10.16 scrub ok 2024-03-04T16:20:51.852 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:51.954 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:51.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:51.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:52.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:52.120 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:52.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:52.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:52.365 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:52.390 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:52.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:52.492 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.539 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:52.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:52.568 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:20:52.575 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:20:52.640 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:52.667 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 10.4 scrub starts 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 3.5 deep-scrub starts 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 3.5 deep-scrub ok 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 10.4 scrub ok 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 969 B/s rd, 0 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:20:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 10.4 scrub starts 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 3.5 deep-scrub starts 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 3.5 deep-scrub ok 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 10.4 scrub ok 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 969 B/s rd, 0 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:20:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:52 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:20:52.770 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:52.799 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:52.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 10.4 scrub starts 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 3.5 deep-scrub starts 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 3.5 deep-scrub ok 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 10.4 scrub ok 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: pgmap v839: 158 pgs: 2 peering, 1 active+remapped+backfilling, 152 active+clean, 3 active+remapped+backfill_toofull; 1.1 GiB data, 8.5 GiB used, 528 GiB / 536 GiB avail; 969 B/s rd, 0 op/s; 2065/54624 objects misplaced (3.780%) 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:20:52.805 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:52 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:257445 10.16 10:68093d2b:::benchmark_data_smithi038_155391_object121488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:52.850265+0000", 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0043861610000000004, 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044182290000000001, 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:52.856 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 257445 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850265+0000", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850265+0000", 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850267+0000", 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8870000000000001e-06 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850273+0000", 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7830000000000004e-06 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:52.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850290+0000", 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6934000000000001e-05 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850300+0000", 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.6600000000000007e-06 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850362+0000", 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2081000000000004e-05 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850453+0000", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0863000000000001e-05 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.850520+0000", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7727999999999999e-05 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853019+0000", 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002498709 2024-03-04T16:20:52.860 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:257448 10.6 10:65979d76:::benchmark_data_smithi038_155391_object121491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:52.851669+0000", 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0029829180000000002, 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031027799999999999, 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:52.861 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 257448 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851669+0000", 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851669+0000", 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851670+0000", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1230000000000001e-06 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851674+0000", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.685e-06 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851686+0000", 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2062000000000001e-05 2024-03-04T16:20:52.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851694+0000", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.447e-06 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851747+0000", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3578000000000001e-05 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851879+0000", 2024-03-04T16:20:52.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131538 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.851957+0000", 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7542999999999999e-05 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854608+0000", 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002651133 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.35585.0:257450 10.10 e391/370 10:0931f4c6:::benchmark_data_smithi038_155391_object121493:head v 391'11619, mlcod=391'11617)", 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:52.853060+0000", 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.0015919020000000001, 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017923819999999999, 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853060+0000", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853060+0000", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853061+0000", 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3289999999999999e-06 2024-03-04T16:20:52.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853068+0000", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9059999999999996e-06 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853076+0000", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1969999999999996e-06 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853082+0000", 2024-03-04T16:20:52.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4559999999999997e-06 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853151+0000", 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8983e-05 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.853173+0000", 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1722999999999999e-05 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_repop(client.35585.0:257444 10.a e391/390 10:5f6eaf73:::benchmark_data_smithi038_155391_object121487:head v 391'15951, mlcod=391'15948)", 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:52.854324+0000", 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.00032742900000000001, 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00058757700000000004, 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "started", 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854324+0000", 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854324+0000", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854325+0000", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.75e-07 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854329+0000", 2024-03-04T16:20:52.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6280000000000004e-06 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854335+0000", 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4480000000000002e-06 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854340+0000", 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1240000000000004e-06 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:52.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854590+0000", 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025000400000000001 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:52.854609+0000", 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9619e-05 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 4 2024-03-04T16:20:52.873 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:20:52.901 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:52.929 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:52.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:53.030 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:53.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:53.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:53.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:53.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:53.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:53.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:20:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.487 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:53.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:53.566 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:20:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: from='client.35789 v1:172.21.15.38:0/416487901' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: 10.8 deep-scrub ok 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:20:53.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: from='client.35789 v1:172.21.15.38:0/416487901' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: 10.8 deep-scrub ok 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:53.568 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:53 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:53.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: from='client.35789 v1:172.21.15.38:0/416487901' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: 10.8 deep-scrub ok 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:20:53.601 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:20:53.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:20:53.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:20:53.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:20:53.602 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:20:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:20:53.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:20:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.640 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:53.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:53.741 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:53.770 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:53.872 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:53.908 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:53.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:53.916 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:54.010 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:54.037 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:54.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:54.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:54.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:54.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:54.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.447 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:54.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:54.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:54.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:54 smithi176 ceph-mon[33619]: 10.b deep-scrub starts 2024-03-04T16:20:54.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:54 smithi176 ceph-mon[33619]: pgmap v840: 158 pgs: 2 peering, 153 active+clean, 3 active+remapped+backfill_toofull; 1.0 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s; 1902/50196 objects misplaced (3.789%) 2024-03-04T16:20:54.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:54.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:54.814 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:54.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:54.959 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:54.985 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:54.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:54 smithi038 ceph-mon[29779]: 10.b deep-scrub starts 2024-03-04T16:20:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:54 smithi038 ceph-mon[29779]: pgmap v840: 158 pgs: 2 peering, 153 active+clean, 3 active+remapped+backfill_toofull; 1.0 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s; 1902/50196 objects misplaced (3.789%) 2024-03-04T16:20:54.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:54 smithi038 ceph-mon[38113]: 10.b deep-scrub starts 2024-03-04T16:20:54.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:54 smithi038 ceph-mon[38113]: pgmap v840: 158 pgs: 2 peering, 153 active+clean, 3 active+remapped+backfill_toofull; 1.0 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.6 KiB/s rd, 1 op/s; 1902/50196 objects misplaced (3.789%) 2024-03-04T16:20:55.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:55.114 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:55.216 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:55.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:55.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.386 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:55.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:55.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:55.511 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:55.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:55.650 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:55.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:55.786 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:55.847 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:20:55.848 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:20:55.848 INFO:teuthology.orchestra.run.smithi176.stdout: "complaint_time": 30, 2024-03-04T16:20:55.848 INFO:teuthology.orchestra.run.smithi176.stdout: "num_blocked_ops": 0 2024-03-04T16:20:55.848 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:20:55.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:55.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:55.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:56.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.042 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:56.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:56.144 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:56.168 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:56.270 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:56.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:56.412 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:56.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:20:56.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:56.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:56.700 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:56.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:56.809 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:20:56.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:56.875 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:56.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:56.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:56.977 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:57.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: pgmap v841: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:57.021 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:56 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:20:57.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:57.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:57.232 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:20:57.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:20:57.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: pgmap v841: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: pgmap v841: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:57.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:56 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:20:57.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:20:57.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:57.385 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:57.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:57.528 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:57.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:57.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:57.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:57.784 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:57.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:20:57.925 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:57.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:20:58.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:58.053 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:20:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:57 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:20:58.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:58.180 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:20:58.180 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:20:58.181 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:20:58.181 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:20:58.181 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:20:58.181 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:20:58.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:58.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:20:58.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:20:58.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:20:58.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:20:58.184 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:20:58.184 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:57 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:20:58.228 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":839,"stamp":"2024-03-04T16:20:51.688694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1180834166,"num_objects":18208,"num_object_clones":0,"num_object_copies":54624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2065,"num_objects_unfound":0,"num_objects_dirty":18208,"num_whiteouts":0,"num_read":2419,"num_read_kb":2265,"num_write":158735,"num_write_kb":5163678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36085,"num_bytes_recovered":2342271824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97594,"ondisk_log_size":97594,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":485,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":8936164,"kb_used_data":3899732,"kb_used_omap":9,"kb_used_meta":5036342,"kb_avail":553534748,"statfs":{"total":575970213888,"available":566819581952,"internally_reserved":0,"allocated":3993325568,"data_stored":3989819232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9561,"internal_metadata":5157214887},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,5,6,3,0,1,2,2,12,15],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-240648192,"num_objects":-3672,"num_object_clones":0,"num_object_copies":-11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2895,"num_objects_unfound":0,"num_objects_dirty":-3672,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":-4234,"num_write_kb":-271232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2454,"num_bytes_recovered":-160890856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.283398"},"pg_stats":[{"pgid":"10.16","version":"389'11651","reported_seq":19232,"reported_epoch":389,"state":"active+remapped+backfilling","last_fresh":"2024-03-04T16:20:48.105176+0000","last_change":"2024-03-04T16:20:45.478501+0000","last_active":"2024-03-04T16:20:48.105176+0000","last_peered":"2024-03-04T16:20:48.105176+0000","last_clean":"2024-03-04T16:20:06.295436+0000","last_became_active":"2024-03-04T16:20:08.765636+0000","last_became_peered":"2024-03-04T16:20:08.765636+0000","last_unstale":"2024-03-04T16:20:48.105176+0000","last_undegraded":"2024-03-04T16:20:48.105176+0000","last_fullsized":"2024-03-04T16:20:48.105176+0000","mapping_epoch":365,"log_start":"324'7559","ondisk_log_start":"324'7559","created":320,"last_epoch_clean":345,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4092,"log_dups_size":0,"ondisk_log_size":4092,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38567936,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":271648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3669,"num_bytes_recovered":240451584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,4],"avail_no_missing":["5","0","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":589}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":60,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414546+0000","last_change":"2024-03-04T16:20:07.476470+0000","last_active":"2024-03-04T16:20:47.414546+0000","last_peered":"2024-03-04T16:20:47.414546+0000","last_clean":"2024-03-04T16:20:47.414546+0000","last_became_active":"2024-03-04T16:20:07.476108+0000","last_became_peered":"2024-03-04T16:20:07.476108+0000","last_unstale":"2024-03-04T16:20:47.414546+0000","last_undegraded":"2024-03-04T16:20:47.414546+0000","last_fullsized":"2024-03-04T16:20:47.414546+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:48.504543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:16:48.504543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425547+0000","last_change":"2024-03-04T16:17:48.323042+0000","last_active":"2024-03-04T16:20:48.425547+0000","last_peered":"2024-03-04T16:20:48.425547+0000","last_clean":"2024-03-04T16:20:48.425547+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:20:48.425547+0000","last_undegraded":"2024-03-04T16:20:48.425547+0000","last_fullsized":"2024-03-04T16:20:48.425547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.322908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.322908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":573,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423591+0000","last_change":"2024-03-04T16:19:04.848699+0000","last_active":"2024-03-04T16:20:48.423591+0000","last_peered":"2024-03-04T16:20:48.423591+0000","last_clean":"2024-03-04T16:20:48.423591+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:20:48.423591+0000","last_undegraded":"2024-03-04T16:20:48.423591+0000","last_fullsized":"2024-03-04T16:20:48.423591+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.623578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.623578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"240'15","reported_seq":498,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425778+0000","last_change":"2024-03-04T16:20:07.544668+0000","last_active":"2024-03-04T16:20:48.425778+0000","last_peered":"2024-03-04T16:20:48.425778+0000","last_clean":"2024-03-04T16:20:48.425778+0000","last_became_active":"2024-03-04T16:20:07.480854+0000","last_became_peered":"2024-03-04T16:20:07.480854+0000","last_unstale":"2024-03-04T16:20:48.425778+0000","last_undegraded":"2024-03-04T16:20:48.425778+0000","last_fullsized":"2024-03-04T16:20:48.425778+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:17:07.218848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.218848+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.17","version":"390'11636","reported_seq":21177,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:49.105783+0000","last_change":"2024-03-04T16:20:49.105862+0000","last_active":"2024-03-04T16:20:49.105783+0000","last_peered":"2024-03-04T16:20:49.105783+0000","last_clean":"2024-03-04T16:20:49.105783+0000","last_became_active":"2024-03-04T16:20:48.432164+0000","last_became_peered":"2024-03-04T16:20:48.432164+0000","last_unstale":"2024-03-04T16:20:49.105783+0000","last_undegraded":"2024-03-04T16:20:49.105783+0000","last_fullsized":"2024-03-04T16:20:49.105783+0000","mapping_epoch":389,"log_start":"327'8142","ondisk_log_start":"327'8142","created":320,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"390'11636","last_scrub_stamp":"2024-03-04T16:20:49.105709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:49.105709+0000","objects_scrubbed":570,"log_size":3494,"log_dups_size":0,"ondisk_log_size":3494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:22:12.876299+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":696,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423356+0000","last_change":"2024-03-04T16:16:55.868681+0000","last_active":"2024-03-04T16:20:48.423356+0000","last_peered":"2024-03-04T16:20:48.423356+0000","last_clean":"2024-03-04T16:20:48.423356+0000","last_became_active":"2024-03-04T16:10:29.246838+0000","last_became_peered":"2024-03-04T16:10:29.246838+0000","last_unstale":"2024-03-04T16:20:48.423356+0000","last_undegraded":"2024-03-04T16:20:48.423356+0000","last_fullsized":"2024-03-04T16:20:48.423356+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.868581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.868581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":662,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425438+0000","last_change":"2024-03-04T16:20:07.487594+0000","last_active":"2024-03-04T16:20:48.425438+0000","last_peered":"2024-03-04T16:20:48.425438+0000","last_clean":"2024-03-04T16:20:48.425438+0000","last_became_active":"2024-03-04T16:20:07.482651+0000","last_became_peered":"2024-03-04T16:20:07.482651+0000","last_unstale":"2024-03-04T16:20:48.425438+0000","last_undegraded":"2024-03-04T16:20:48.425438+0000","last_fullsized":"2024-03-04T16:20:48.425438+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.629592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":714,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423104+0000","last_change":"2024-03-04T16:19:04.779817+0000","last_active":"2024-03-04T16:20:48.423104+0000","last_peered":"2024-03-04T16:20:48.423104+0000","last_clean":"2024-03-04T16:20:48.423104+0000","last_became_active":"2024-03-04T16:19:04.779498+0000","last_became_peered":"2024-03-04T16:19:04.779498+0000","last_unstale":"2024-03-04T16:20:48.423104+0000","last_undegraded":"2024-03-04T16:20:48.423104+0000","last_fullsized":"2024-03-04T16:20:48.423104+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:07.030580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:30.752256+0000","last_clean_scrub_stamp":"2024-03-04T16:16:07.030580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"240'27","reported_seq":829,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422644+0000","last_change":"2024-03-04T16:19:04.864691+0000","last_active":"2024-03-04T16:20:48.422644+0000","last_peered":"2024-03-04T16:20:48.422644+0000","last_clean":"2024-03-04T16:20:48.422644+0000","last_became_active":"2024-03-04T16:19:04.778671+0000","last_became_peered":"2024-03-04T16:19:04.778671+0000","last_unstale":"2024-03-04T16:20:48.422644+0000","last_undegraded":"2024-03-04T16:20:48.422644+0000","last_fullsized":"2024-03-04T16:20:48.422644+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:15:51.042010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.042010+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.14","version":"390'11701","reported_seq":24765,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.502362+0000","last_change":"2024-03-04T16:20:37.172641+0000","last_active":"2024-03-04T16:20:48.502362+0000","last_peered":"2024-03-04T16:20:48.502362+0000","last_clean":"2024-03-04T16:20:48.502362+0000","last_became_active":"2024-03-04T16:20:25.387362+0000","last_became_peered":"2024-03-04T16:20:25.387362+0000","last_unstale":"2024-03-04T16:20:48.502362+0000","last_undegraded":"2024-03-04T16:20:48.502362+0000","last_fullsized":"2024-03-04T16:20:48.502362+0000","mapping_epoch":380,"log_start":"328'8470","ondisk_log_start":"328'8470","created":320,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"385'11346","last_scrub_stamp":"2024-03-04T16:20:37.172416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:37.172416+0000","objects_scrubbed":885,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:47.447161+0000","scrub_duration":369,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":562,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425121+0000","last_change":"2024-03-04T16:18:12.891866+0000","last_active":"2024-03-04T16:20:48.425121+0000","last_peered":"2024-03-04T16:20:48.425121+0000","last_clean":"2024-03-04T16:20:48.425121+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:20:48.425121+0000","last_undegraded":"2024-03-04T16:20:48.425121+0000","last_fullsized":"2024-03-04T16:20:48.425121+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:18:12.891748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":699,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415234+0000","last_change":"2024-03-04T16:17:03.584844+0000","last_active":"2024-03-04T16:20:47.415234+0000","last_peered":"2024-03-04T16:20:47.415234+0000","last_clean":"2024-03-04T16:20:47.415234+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:20:47.415234+0000","last_undegraded":"2024-03-04T16:20:47.415234+0000","last_fullsized":"2024-03-04T16:20:47.415234+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.584766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:20.132710+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.584766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":727,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415614+0000","last_change":"2024-03-04T16:20:07.479810+0000","last_active":"2024-03-04T16:20:47.415614+0000","last_peered":"2024-03-04T16:20:47.415614+0000","last_clean":"2024-03-04T16:20:47.415614+0000","last_became_active":"2024-03-04T16:20:07.479397+0000","last_became_peered":"2024-03-04T16:20:07.479397+0000","last_unstale":"2024-03-04T16:20:47.415614+0000","last_undegraded":"2024-03-04T16:20:47.415614+0000","last_fullsized":"2024-03-04T16:20:47.415614+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.732730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.732730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"240'19","reported_seq":518,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423717+0000","last_change":"2024-03-04T16:20:07.546316+0000","last_active":"2024-03-04T16:20:48.423717+0000","last_peered":"2024-03-04T16:20:48.423717+0000","last_clean":"2024-03-04T16:20:48.423717+0000","last_became_active":"2024-03-04T16:20:07.491874+0000","last_became_peered":"2024-03-04T16:20:07.491874+0000","last_unstale":"2024-03-04T16:20:48.423717+0000","last_undegraded":"2024-03-04T16:20:48.423717+0000","last_fullsized":"2024-03-04T16:20:48.423717+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:49.276987+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.276987+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.15","version":"390'11822","reported_seq":23845,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.513186+0000","last_change":"2024-03-04T16:20:37.256411+0000","last_active":"2024-03-04T16:20:48.513186+0000","last_peered":"2024-03-04T16:20:48.513186+0000","last_clean":"2024-03-04T16:20:48.513186+0000","last_became_active":"2024-03-04T16:19:00.357537+0000","last_became_peered":"2024-03-04T16:19:00.357537+0000","last_unstale":"2024-03-04T16:20:48.513186+0000","last_undegraded":"2024-03-04T16:20:48.513186+0000","last_fullsized":"2024-03-04T16:20:48.513186+0000","mapping_epoch":335,"log_start":"328'8299","ondisk_log_start":"328'8299","created":320,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":5,"last_scrub":"385'11512","last_scrub_stamp":"2024-03-04T16:20:37.256197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:37.256197+0000","objects_scrubbed":845,"log_size":3523,"log_dups_size":0,"ondisk_log_size":3523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:22:01.506939+0000","scrub_duration":427,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":319,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":703,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414746+0000","last_change":"2024-03-04T16:16:09.872300+0000","last_active":"2024-03-04T16:20:47.414746+0000","last_peered":"2024-03-04T16:20:47.414746+0000","last_clean":"2024-03-04T16:20:47.414746+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:20:47.414746+0000","last_undegraded":"2024-03-04T16:20:47.414746+0000","last_fullsized":"2024-03-04T16:20:47.414746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:16:09.872152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":722,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413976+0000","last_change":"2024-03-04T16:20:07.494466+0000","last_active":"2024-03-04T16:20:47.413976+0000","last_peered":"2024-03-04T16:20:47.413976+0000","last_clean":"2024-03-04T16:20:47.413976+0000","last_became_active":"2024-03-04T16:20:07.494164+0000","last_became_peered":"2024-03-04T16:20:07.494164+0000","last_unstale":"2024-03-04T16:20:47.413976+0000","last_undegraded":"2024-03-04T16:20:47.413976+0000","last_fullsized":"2024-03-04T16:20:47.413976+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:17.593078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:19:17.593078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":713,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415318+0000","last_change":"2024-03-04T16:20:07.468848+0000","last_active":"2024-03-04T16:20:47.415318+0000","last_peered":"2024-03-04T16:20:47.415318+0000","last_clean":"2024-03-04T16:20:47.415318+0000","last_became_active":"2024-03-04T16:20:07.468521+0000","last_became_peered":"2024-03-04T16:20:07.468521+0000","last_unstale":"2024-03-04T16:20:47.415318+0000","last_undegraded":"2024-03-04T16:20:47.415318+0000","last_fullsized":"2024-03-04T16:20:47.415318+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:06.843231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:06.843231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"240'9","reported_seq":492,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423071+0000","last_change":"2024-03-04T16:20:07.522036+0000","last_active":"2024-03-04T16:20:48.423071+0000","last_peered":"2024-03-04T16:20:48.423071+0000","last_clean":"2024-03-04T16:20:48.423071+0000","last_became_active":"2024-03-04T16:20:07.482406+0000","last_became_peered":"2024-03-04T16:20:07.482406+0000","last_unstale":"2024-03-04T16:20:48.423071+0000","last_undegraded":"2024-03-04T16:20:48.423071+0000","last_fullsized":"2024-03-04T16:20:48.423071+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'9","last_scrub_stamp":"2024-03-04T16:17:54.411724+0000","last_deep_scrub":"170'7","last_deep_scrub_stamp":"2024-03-04T16:12:15.539528+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.411724+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"390'11723","reported_seq":24791,"reported_epoch":390,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:20:48.435922+0000","last_change":"2024-03-04T16:20:35.993009+0000","last_active":"2024-03-04T16:20:48.435922+0000","last_peered":"2024-03-04T16:20:48.435922+0000","last_clean":"2024-03-04T16:18:20.339570+0000","last_became_active":"2024-03-04T16:19:04.891568+0000","last_became_peered":"2024-03-04T16:19:04.891568+0000","last_unstale":"2024-03-04T16:20:48.435922+0000","last_undegraded":"2024-03-04T16:20:48.435922+0000","last_fullsized":"2024-03-04T16:20:48.435922+0000","mapping_epoch":337,"log_start":"327'8268","ondisk_log_start":"327'8268","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3455,"log_dups_size":0,"ondisk_log_size":3455,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39354368,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":825,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":273824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3254,"num_bytes_recovered":213254144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[0,1,4],"avail_no_missing":["0","1","3","4"],"object_location_counts":[{"shards":"0,1,4","objects":601}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":337,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.416099+0000","last_change":"2024-03-04T16:20:07.485615+0000","last_active":"2024-03-04T16:20:47.416099+0000","last_peered":"2024-03-04T16:20:47.416099+0000","last_clean":"2024-03-04T16:20:47.416099+0000","last_became_active":"2024-03-04T16:20:07.485229+0000","last_became_peered":"2024-03-04T16:20:07.485229+0000","last_unstale":"2024-03-04T16:20:47.416099+0000","last_undegraded":"2024-03-04T16:20:47.416099+0000","last_fullsized":"2024-03-04T16:20:47.416099+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:59.573389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:59.573389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":716,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426551+0000","last_change":"2024-03-04T16:20:07.498509+0000","last_active":"2024-03-04T16:20:48.426551+0000","last_peered":"2024-03-04T16:20:48.426551+0000","last_clean":"2024-03-04T16:20:48.426551+0000","last_became_active":"2024-03-04T16:20:07.497650+0000","last_became_peered":"2024-03-04T16:20:07.497650+0000","last_unstale":"2024-03-04T16:20:48.426551+0000","last_undegraded":"2024-03-04T16:20:48.426551+0000","last_fullsized":"2024-03-04T16:20:48.426551+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:49.317160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:49.317160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":560,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424120+0000","last_change":"2024-03-04T16:20:07.498978+0000","last_active":"2024-03-04T16:20:48.424120+0000","last_peered":"2024-03-04T16:20:48.424120+0000","last_clean":"2024-03-04T16:20:48.424120+0000","last_became_active":"2024-03-04T16:20:07.498539+0000","last_became_peered":"2024-03-04T16:20:07.498539+0000","last_unstale":"2024-03-04T16:20:48.424120+0000","last_undegraded":"2024-03-04T16:20:48.424120+0000","last_fullsized":"2024-03-04T16:20:48.424120+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:04.653768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:04.653768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"286'34","reported_seq":129,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413463+0000","last_change":"2024-03-04T16:19:05.017701+0000","last_active":"2024-03-04T16:20:47.413463+0000","last_peered":"2024-03-04T16:20:47.413463+0000","last_clean":"2024-03-04T16:20:47.413463+0000","last_became_active":"2024-03-04T16:19:04.771891+0000","last_became_peered":"2024-03-04T16:19:04.771891+0000","last_unstale":"2024-03-04T16:20:47.413463+0000","last_undegraded":"2024-03-04T16:20:47.413463+0000","last_fullsized":"2024-03-04T16:20:47.413463+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:18:15.488262+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.488262+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.13","version":"390'11705","reported_seq":20087,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.491814+0000","last_change":"2024-03-04T16:20:38.117859+0000","last_active":"2024-03-04T16:20:48.491814+0000","last_peered":"2024-03-04T16:20:48.491814+0000","last_clean":"2024-03-04T16:20:48.491814+0000","last_became_active":"2024-03-04T16:20:04.404168+0000","last_became_peered":"2024-03-04T16:20:04.404168+0000","last_unstale":"2024-03-04T16:20:48.491814+0000","last_undegraded":"2024-03-04T16:20:48.491814+0000","last_fullsized":"2024-03-04T16:20:48.491814+0000","mapping_epoch":362,"log_start":"328'8396","ondisk_log_start":"328'8396","created":320,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":5,"last_scrub":"385'11413","last_scrub_stamp":"2024-03-04T16:20:38.117659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:38.117659+0000","objects_scrubbed":872,"log_size":3309,"log_dups_size":0,"ondisk_log_size":3309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:59.789603+0000","scrub_duration":336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":697,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428260+0000","last_change":"2024-03-04T16:17:48.459727+0000","last_active":"2024-03-04T16:20:48.428260+0000","last_peered":"2024-03-04T16:20:48.428260+0000","last_clean":"2024-03-04T16:20:48.428260+0000","last_became_active":"2024-03-04T16:10:29.351698+0000","last_became_peered":"2024-03-04T16:10:29.351698+0000","last_unstale":"2024-03-04T16:20:48.428260+0000","last_undegraded":"2024-03-04T16:20:48.428260+0000","last_fullsized":"2024-03-04T16:20:48.428260+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:48.459594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:17:48.459594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":662,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424763+0000","last_change":"2024-03-04T16:20:07.474700+0000","last_active":"2024-03-04T16:20:48.424763+0000","last_peered":"2024-03-04T16:20:48.424763+0000","last_clean":"2024-03-04T16:20:48.424763+0000","last_became_active":"2024-03-04T16:20:07.474386+0000","last_became_peered":"2024-03-04T16:20:07.474386+0000","last_unstale":"2024-03-04T16:20:48.424763+0000","last_undegraded":"2024-03-04T16:20:48.424763+0000","last_fullsized":"2024-03-04T16:20:48.424763+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.414689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.414689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":670,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426095+0000","last_change":"2024-03-04T16:20:07.506874+0000","last_active":"2024-03-04T16:20:48.426095+0000","last_peered":"2024-03-04T16:20:48.426095+0000","last_clean":"2024-03-04T16:20:48.426095+0000","last_became_active":"2024-03-04T16:20:07.503831+0000","last_became_peered":"2024-03-04T16:20:07.503831+0000","last_unstale":"2024-03-04T16:20:48.426095+0000","last_undegraded":"2024-03-04T16:20:48.426095+0000","last_fullsized":"2024-03-04T16:20:48.426095+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:17.538492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:17.538492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"240'18","reported_seq":756,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424054+0000","last_change":"2024-03-04T16:20:07.540299+0000","last_active":"2024-03-04T16:20:48.424054+0000","last_peered":"2024-03-04T16:20:48.424054+0000","last_clean":"2024-03-04T16:20:48.424054+0000","last_became_active":"2024-03-04T16:20:07.483051+0000","last_became_peered":"2024-03-04T16:20:07.483051+0000","last_unstale":"2024-03-04T16:20:48.424054+0000","last_undegraded":"2024-03-04T16:20:48.424054+0000","last_fullsized":"2024-03-04T16:20:48.424054+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:17:03.702661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.702661+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.10","version":"389'11521","reported_seq":20933,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.689231+0000","last_change":"2024-03-04T16:20:36.556248+0000","last_active":"2024-03-04T16:20:47.689231+0000","last_peered":"2024-03-04T16:20:47.689231+0000","last_clean":"2024-03-04T16:20:47.689231+0000","last_became_active":"2024-03-04T16:20:13.539873+0000","last_became_peered":"2024-03-04T16:20:13.539873+0000","last_unstale":"2024-03-04T16:20:47.689231+0000","last_undegraded":"2024-03-04T16:20:47.689231+0000","last_fullsized":"2024-03-04T16:20:47.689231+0000","mapping_epoch":370,"log_start":"324'7700","ondisk_log_start":"324'7700","created":320,"last_epoch_clean":371,"parent":"0.0","parent_split_bits":5,"last_scrub":"385'11212","last_scrub_stamp":"2024-03-04T16:20:36.556136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:36.556136+0000","objects_scrubbed":842,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:22:00.754477+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":726,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414474+0000","last_change":"2024-03-04T16:19:06.576246+0000","last_active":"2024-03-04T16:20:47.414474+0000","last_peered":"2024-03-04T16:20:47.414474+0000","last_clean":"2024-03-04T16:20:47.414474+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:20:47.414474+0000","last_undegraded":"2024-03-04T16:20:47.414474+0000","last_fullsized":"2024-03-04T16:20:47.414474+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:19:06.576128+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:19:06.576128+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"366'12","reported_seq":266,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414708+0000","last_change":"2024-03-04T16:20:07.491623+0000","last_active":"2024-03-04T16:20:47.414708+0000","last_peered":"2024-03-04T16:20:47.414708+0000","last_clean":"2024-03-04T16:20:47.414708+0000","last_became_active":"2024-03-04T16:20:07.480474+0000","last_became_peered":"2024-03-04T16:20:07.480474+0000","last_unstale":"2024-03-04T16:20:47.414708+0000","last_undegraded":"2024-03-04T16:20:47.414708+0000","last_fullsized":"2024-03-04T16:20:47.414708+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'10","last_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:17:08.205209+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":729,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425713+0000","last_change":"2024-03-04T16:20:07.510265+0000","last_active":"2024-03-04T16:20:48.425713+0000","last_peered":"2024-03-04T16:20:48.425713+0000","last_clean":"2024-03-04T16:20:48.425713+0000","last_became_active":"2024-03-04T16:20:07.473621+0000","last_became_peered":"2024-03-04T16:20:07.473621+0000","last_unstale":"2024-03-04T16:20:48.425713+0000","last_undegraded":"2024-03-04T16:20:48.425713+0000","last_fullsized":"2024-03-04T16:20:48.425713+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:18:16.465669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.465669+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"240'20","reported_seq":582,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425100+0000","last_change":"2024-03-04T16:16:08.556672+0000","last_active":"2024-03-04T16:20:48.425100+0000","last_peered":"2024-03-04T16:20:48.425100+0000","last_clean":"2024-03-04T16:20:48.425100+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:20:48.425100+0000","last_undegraded":"2024-03-04T16:20:48.425100+0000","last_fullsized":"2024-03-04T16:20:48.425100+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:08.553910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.553910+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"389'13955","reported_seq":22009,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.032942+0000","last_change":"2024-03-04T16:19:36.899132+0000","last_active":"2024-03-04T16:20:48.032942+0000","last_peered":"2024-03-04T16:20:48.032942+0000","last_clean":"2024-03-04T16:20:48.032942+0000","last_became_active":"2024-03-04T16:18:53.406638+0000","last_became_peered":"2024-03-04T16:18:53.406638+0000","last_unstale":"2024-03-04T16:20:48.032942+0000","last_undegraded":"2024-03-04T16:20:48.032942+0000","last_fullsized":"2024-03-04T16:20:48.032942+0000","mapping_epoch":331,"log_start":"349'9896","ondisk_log_start":"349'9896","created":320,"last_epoch_clean":332,"parent":"0.0","parent_split_bits":5,"last_scrub":"352'12217","last_scrub_stamp":"2024-03-04T16:19:36.899006+0000","last_deep_scrub":"352'12217","last_deep_scrub_stamp":"2024-03-04T16:19:36.899006+0000","last_clean_scrub_stamp":"2024-03-04T16:19:36.899006+0000","objects_scrubbed":2452,"log_size":4059,"log_dups_size":0,"ondisk_log_size":4059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"queued for scrub","scrub_duration":14516,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35586048,"num_objects":543,"num_object_clones":0,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1909,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":704,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426698+0000","last_change":"2024-03-04T16:20:07.478257+0000","last_active":"2024-03-04T16:20:48.426698+0000","last_peered":"2024-03-04T16:20:48.426698+0000","last_clean":"2024-03-04T16:20:48.426698+0000","last_became_active":"2024-03-04T16:20:07.477199+0000","last_became_peered":"2024-03-04T16:20:07.477199+0000","last_unstale":"2024-03-04T16:20:48.426698+0000","last_undegraded":"2024-03-04T16:20:48.426698+0000","last_fullsized":"2024-03-04T16:20:48.426698+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:03.492000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:03.492000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":231,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424677+0000","last_change":"2024-03-04T16:19:04.848756+0000","last_active":"2024-03-04T16:20:48.424677+0000","last_peered":"2024-03-04T16:20:48.424677+0000","last_clean":"2024-03-04T16:20:48.424677+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:20:48.424677+0000","last_undegraded":"2024-03-04T16:20:48.424677+0000","last_fullsized":"2024-03-04T16:20:48.424677+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:00.852665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.852665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":714,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422342+0000","last_change":"2024-03-04T16:16:08.076532+0000","last_active":"2024-03-04T16:20:48.422342+0000","last_peered":"2024-03-04T16:20:48.422342+0000","last_clean":"2024-03-04T16:20:48.422342+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:20:48.422342+0000","last_undegraded":"2024-03-04T16:20:48.422342+0000","last_fullsized":"2024-03-04T16:20:48.422342+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:16:08.076402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"240'19","reported_seq":490,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415950+0000","last_change":"2024-03-04T16:17:06.624302+0000","last_active":"2024-03-04T16:20:47.415950+0000","last_peered":"2024-03-04T16:20:47.415950+0000","last_clean":"2024-03-04T16:20:47.415950+0000","last_became_active":"2024-03-04T16:10:29.352643+0000","last_became_peered":"2024-03-04T16:10:29.352643+0000","last_unstale":"2024-03-04T16:20:47.415950+0000","last_undegraded":"2024-03-04T16:20:47.415950+0000","last_fullsized":"2024-03-04T16:20:47.415950+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:06.624154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.624154+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":730,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414006+0000","last_change":"2024-03-04T16:20:07.478055+0000","last_active":"2024-03-04T16:20:47.414006+0000","last_peered":"2024-03-04T16:20:47.414006+0000","last_clean":"2024-03-04T16:20:47.414006+0000","last_became_active":"2024-03-04T16:20:07.477661+0000","last_became_peered":"2024-03-04T16:20:07.477661+0000","last_unstale":"2024-03-04T16:20:47.414006+0000","last_undegraded":"2024-03-04T16:20:47.414006+0000","last_fullsized":"2024-03-04T16:20:47.414006+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:05.623916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:19:05.623916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":62,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413598+0000","last_change":"2024-03-04T16:20:07.484961+0000","last_active":"2024-03-04T16:20:47.413598+0000","last_peered":"2024-03-04T16:20:47.413598+0000","last_clean":"2024-03-04T16:20:47.413598+0000","last_became_active":"2024-03-04T16:20:07.484641+0000","last_became_peered":"2024-03-04T16:20:07.484641+0000","last_unstale":"2024-03-04T16:20:47.413598+0000","last_undegraded":"2024-03-04T16:20:47.413598+0000","last_fullsized":"2024-03-04T16:20:47.413598+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:58.573599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:15:58.573599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":554,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423498+0000","last_change":"2024-03-04T16:17:58.626787+0000","last_active":"2024-03-04T16:20:48.423498+0000","last_peered":"2024-03-04T16:20:48.423498+0000","last_clean":"2024-03-04T16:20:48.423498+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:20:48.423498+0000","last_undegraded":"2024-03-04T16:20:48.423498+0000","last_fullsized":"2024-03-04T16:20:48.423498+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:58.626660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.626660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"240'8","reported_seq":734,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422448+0000","last_change":"2024-03-04T16:20:07.506301+0000","last_active":"2024-03-04T16:20:48.422448+0000","last_peered":"2024-03-04T16:20:48.422448+0000","last_clean":"2024-03-04T16:20:48.422448+0000","last_became_active":"2024-03-04T16:20:07.481769+0000","last_became_peered":"2024-03-04T16:20:07.481769+0000","last_unstale":"2024-03-04T16:20:48.422448+0000","last_undegraded":"2024-03-04T16:20:48.422448+0000","last_fullsized":"2024-03-04T16:20:48.422448+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:16:59.675176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.675176+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":714,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423887+0000","last_change":"2024-03-04T16:19:04.781577+0000","last_active":"2024-03-04T16:20:48.423887+0000","last_peered":"2024-03-04T16:20:48.423887+0000","last_clean":"2024-03-04T16:20:48.423887+0000","last_became_active":"2024-03-04T16:19:04.781224+0000","last_became_peered":"2024-03-04T16:19:04.781224+0000","last_unstale":"2024-03-04T16:20:48.423887+0000","last_undegraded":"2024-03-04T16:20:48.423887+0000","last_fullsized":"2024-03-04T16:20:48.423887+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:14.574861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.574861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":720,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413884+0000","last_change":"2024-03-04T16:20:07.486138+0000","last_active":"2024-03-04T16:20:47.413884+0000","last_peered":"2024-03-04T16:20:47.413884+0000","last_clean":"2024-03-04T16:20:47.413884+0000","last_became_active":"2024-03-04T16:20:07.485822+0000","last_became_peered":"2024-03-04T16:20:07.485822+0000","last_unstale":"2024-03-04T16:20:47.413884+0000","last_undegraded":"2024-03-04T16:20:47.413884+0000","last_fullsized":"2024-03-04T16:20:47.413884+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:08.601393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:19:08.601393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":715,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415786+0000","last_change":"2024-03-04T16:17:01.618230+0000","last_active":"2024-03-04T16:20:47.415786+0000","last_peered":"2024-03-04T16:20:47.415786+0000","last_clean":"2024-03-04T16:20:47.415786+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:20:47.415786+0000","last_undegraded":"2024-03-04T16:20:47.415786+0000","last_fullsized":"2024-03-04T16:20:47.415786+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.618146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.618146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"240'19","reported_seq":747,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422512+0000","last_change":"2024-03-04T16:18:06.391143+0000","last_active":"2024-03-04T16:20:48.422512+0000","last_peered":"2024-03-04T16:20:48.422512+0000","last_clean":"2024-03-04T16:20:48.422512+0000","last_became_active":"2024-03-04T16:10:29.339076+0000","last_became_peered":"2024-03-04T16:10:29.339076+0000","last_unstale":"2024-03-04T16:20:48.422512+0000","last_undegraded":"2024-03-04T16:20:48.422512+0000","last_fullsized":"2024-03-04T16:20:48.422512+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:18:06.391045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:06.391045+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":554,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425292+0000","last_change":"2024-03-04T16:18:10.894569+0000","last_active":"2024-03-04T16:20:48.425292+0000","last_peered":"2024-03-04T16:20:48.425292+0000","last_clean":"2024-03-04T16:20:48.425292+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:20:48.425292+0000","last_undegraded":"2024-03-04T16:20:48.425292+0000","last_fullsized":"2024-03-04T16:20:48.425292+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:10.893919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:18:10.893919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":704,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423258+0000","last_change":"2024-03-04T16:16:05.072152+0000","last_active":"2024-03-04T16:20:48.423258+0000","last_peered":"2024-03-04T16:20:48.423258+0000","last_clean":"2024-03-04T16:20:48.423258+0000","last_became_active":"2024-03-04T16:12:23.623551+0000","last_became_peered":"2024-03-04T16:12:23.623551+0000","last_unstale":"2024-03-04T16:20:48.423258+0000","last_undegraded":"2024-03-04T16:20:48.423258+0000","last_fullsized":"2024-03-04T16:20:48.423258+0000","mapping_epoch":233,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:05.071956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:05.071956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":722,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414660+0000","last_change":"2024-03-04T16:20:07.498759+0000","last_active":"2024-03-04T16:20:47.414660+0000","last_peered":"2024-03-04T16:20:47.414660+0000","last_clean":"2024-03-04T16:20:47.414660+0000","last_became_active":"2024-03-04T16:20:07.498410+0000","last_became_peered":"2024-03-04T16:20:07.498410+0000","last_unstale":"2024-03-04T16:20:47.414660+0000","last_undegraded":"2024-03-04T16:20:47.414660+0000","last_fullsized":"2024-03-04T16:20:47.414660+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:56.238364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:56.238364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"240'19","reported_seq":791,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426673+0000","last_change":"2024-03-04T16:19:04.909849+0000","last_active":"2024-03-04T16:20:48.426673+0000","last_peered":"2024-03-04T16:20:48.426673+0000","last_clean":"2024-03-04T16:20:48.426673+0000","last_became_active":"2024-03-04T16:19:04.791277+0000","last_became_peered":"2024-03-04T16:19:04.791277+0000","last_unstale":"2024-03-04T16:20:48.426673+0000","last_undegraded":"2024-03-04T16:20:48.426673+0000","last_fullsized":"2024-03-04T16:20:48.426673+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:17:51.289785+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:17:51.289785+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":702,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422258+0000","last_change":"2024-03-04T16:18:17.376026+0000","last_active":"2024-03-04T16:20:48.422258+0000","last_peered":"2024-03-04T16:20:48.422258+0000","last_clean":"2024-03-04T16:20:48.422258+0000","last_became_active":"2024-03-04T16:07:36.222722+0000","last_became_peered":"2024-03-04T16:07:36.222722+0000","last_unstale":"2024-03-04T16:20:48.422258+0000","last_undegraded":"2024-03-04T16:20:48.422258+0000","last_fullsized":"2024-03-04T16:20:48.422258+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:17.375837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:18:17.375837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":710,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426498+0000","last_change":"2024-03-04T16:17:54.262357+0000","last_active":"2024-03-04T16:20:48.426498+0000","last_peered":"2024-03-04T16:20:48.426498+0000","last_clean":"2024-03-04T16:20:48.426498+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:20:48.426498+0000","last_undegraded":"2024-03-04T16:20:48.426498+0000","last_fullsized":"2024-03-04T16:20:48.426498+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:54.262228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.262228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":668,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414300+0000","last_change":"2024-03-04T16:20:07.494879+0000","last_active":"2024-03-04T16:20:47.414300+0000","last_peered":"2024-03-04T16:20:47.414300+0000","last_clean":"2024-03-04T16:20:47.414300+0000","last_became_active":"2024-03-04T16:20:07.494411+0000","last_became_peered":"2024-03-04T16:20:07.494411+0000","last_unstale":"2024-03-04T16:20:47.414300+0000","last_undegraded":"2024-03-04T16:20:47.414300+0000","last_fullsized":"2024-03-04T16:20:47.414300+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:57.350240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.350240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"316'24","reported_seq":683,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428548+0000","last_change":"2024-03-04T16:20:07.530032+0000","last_active":"2024-03-04T16:20:48.428548+0000","last_peered":"2024-03-04T16:20:48.428548+0000","last_clean":"2024-03-04T16:20:48.428548+0000","last_became_active":"2024-03-04T16:20:07.495681+0000","last_became_peered":"2024-03-04T16:20:07.495681+0000","last_unstale":"2024-03-04T16:20:48.428548+0000","last_undegraded":"2024-03-04T16:20:48.428548+0000","last_fullsized":"2024-03-04T16:20:48.428548+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'24","last_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_deep_scrub":"316'24","last_deep_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_clean_scrub_stamp":"2024-03-04T16:18:04.512502+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":585,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414435+0000","last_change":"2024-03-04T16:19:22.631070+0000","last_active":"2024-03-04T16:20:47.414435+0000","last_peered":"2024-03-04T16:20:47.414435+0000","last_clean":"2024-03-04T16:20:47.414435+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:20:47.414435+0000","last_undegraded":"2024-03-04T16:20:47.414435+0000","last_fullsized":"2024-03-04T16:20:47.414435+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:22.630934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:19:22.630934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:51.708453+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":946,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415715+0000","last_change":"2024-03-04T16:16:59.613637+0000","last_active":"2024-03-04T16:20:47.415715+0000","last_peered":"2024-03-04T16:20:47.415715+0000","last_clean":"2024-03-04T16:20:47.415715+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:20:47.415715+0000","last_undegraded":"2024-03-04T16:20:47.415715+0000","last_fullsized":"2024-03-04T16:20:47.415715+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:16:59.613549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:59.613549+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":770,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413758+0000","last_change":"2024-03-04T16:20:07.504946+0000","last_active":"2024-03-04T16:20:47.413758+0000","last_peered":"2024-03-04T16:20:47.413758+0000","last_clean":"2024-03-04T16:20:47.413758+0000","last_became_active":"2024-03-04T16:20:07.487868+0000","last_became_peered":"2024-03-04T16:20:07.487868+0000","last_unstale":"2024-03-04T16:20:47.413758+0000","last_undegraded":"2024-03-04T16:20:47.413758+0000","last_fullsized":"2024-03-04T16:20:47.413758+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:19:09.596049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:19:09.596049+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"240'20","reported_seq":756,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424097+0000","last_change":"2024-03-04T16:16:53.775738+0000","last_active":"2024-03-04T16:20:48.424097+0000","last_peered":"2024-03-04T16:20:48.424097+0000","last_clean":"2024-03-04T16:20:48.424097+0000","last_became_active":"2024-03-04T16:10:17.483918+0000","last_became_peered":"2024-03-04T16:10:17.483918+0000","last_unstale":"2024-03-04T16:20:48.424097+0000","last_undegraded":"2024-03-04T16:20:48.424097+0000","last_fullsized":"2024-03-04T16:20:48.424097+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:16:53.775609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.775609+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":656,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415228+0000","last_change":"2024-03-04T16:20:07.486667+0000","last_active":"2024-03-04T16:20:47.415228+0000","last_peered":"2024-03-04T16:20:47.415228+0000","last_clean":"2024-03-04T16:20:47.415228+0000","last_became_active":"2024-03-04T16:20:07.486320+0000","last_became_peered":"2024-03-04T16:20:07.486320+0000","last_unstale":"2024-03-04T16:20:47.415228+0000","last_undegraded":"2024-03-04T16:20:47.415228+0000","last_fullsized":"2024-03-04T16:20:47.415228+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.474636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.474636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"338'11","reported_seq":941,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425066+0000","last_change":"2024-03-04T16:19:04.890313+0000","last_active":"2024-03-04T16:20:48.425066+0000","last_peered":"2024-03-04T16:20:48.425066+0000","last_clean":"2024-03-04T16:20:48.425066+0000","last_became_active":"2024-03-04T16:19:04.780989+0000","last_became_peered":"2024-03-04T16:19:04.780989+0000","last_unstale":"2024-03-04T16:20:48.425066+0000","last_undegraded":"2024-03-04T16:20:48.425066+0000","last_fullsized":"2024-03-04T16:20:48.425066+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.365318+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":609,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413545+0000","last_change":"2024-03-04T16:17:47.360887+0000","last_active":"2024-03-04T16:20:47.413545+0000","last_peered":"2024-03-04T16:20:47.413545+0000","last_clean":"2024-03-04T16:20:47.413545+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:20:47.413545+0000","last_undegraded":"2024-03-04T16:20:47.413545+0000","last_fullsized":"2024-03-04T16:20:47.413545+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:17:47.360747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.360747+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"240'15","reported_seq":736,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415952+0000","last_change":"2024-03-04T16:18:01.423163+0000","last_active":"2024-03-04T16:20:47.415952+0000","last_peered":"2024-03-04T16:20:47.415952+0000","last_clean":"2024-03-04T16:20:47.415952+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:20:47.415952+0000","last_undegraded":"2024-03-04T16:20:47.415952+0000","last_fullsized":"2024-03-04T16:20:47.415952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:18:01.423072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:01.423072+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.18","version":"390'11968","reported_seq":22901,"reported_epoch":390,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:20:48.549233+0000","last_change":"2024-03-04T16:20:37.519564+0000","last_active":"2024-03-04T16:20:48.549233+0000","last_peered":"2024-03-04T16:20:48.549233+0000","last_clean":"2024-03-04T16:18:20.343788+0000","last_became_active":"2024-03-04T16:19:04.904088+0000","last_became_peered":"2024-03-04T16:19:04.904088+0000","last_unstale":"2024-03-04T16:20:48.549233+0000","last_undegraded":"2024-03-04T16:20:48.549233+0000","last_fullsized":"2024-03-04T16:20:48.549233+0000","mapping_epoch":337,"log_start":"328'8487","ondisk_log_start":"328'8487","created":320,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35749888,"num_objects":546,"num_object_clones":0,"num_object_copies":1638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":618,"num_objects_unfound":0,"num_objects_dirty":546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8146,"num_write_kb":278112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3566,"num_bytes_recovered":233701376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":546}],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":119,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413613+0000","last_change":"2024-03-04T16:20:07.491864+0000","last_active":"2024-03-04T16:20:47.413613+0000","last_peered":"2024-03-04T16:20:47.413613+0000","last_clean":"2024-03-04T16:20:47.413613+0000","last_became_active":"2024-03-04T16:20:07.491512+0000","last_became_peered":"2024-03-04T16:20:07.491512+0000","last_unstale":"2024-03-04T16:20:47.413613+0000","last_undegraded":"2024-03-04T16:20:47.413613+0000","last_fullsized":"2024-03-04T16:20:47.413613+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.332239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.332239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":694,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428164+0000","last_change":"2024-03-04T16:17:55.208057+0000","last_active":"2024-03-04T16:20:48.428164+0000","last_peered":"2024-03-04T16:20:48.428164+0000","last_clean":"2024-03-04T16:20:48.428164+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:20:48.428164+0000","last_undegraded":"2024-03-04T16:20:48.428164+0000","last_fullsized":"2024-03-04T16:20:48.428164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:55.207977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.207977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":728,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413835+0000","last_change":"2024-03-04T16:20:07.494445+0000","last_active":"2024-03-04T16:20:47.413835+0000","last_peered":"2024-03-04T16:20:47.413835+0000","last_clean":"2024-03-04T16:20:47.413835+0000","last_became_active":"2024-03-04T16:20:07.494101+0000","last_became_peered":"2024-03-04T16:20:07.494101+0000","last_unstale":"2024-03-04T16:20:47.413835+0000","last_undegraded":"2024-03-04T16:20:47.413835+0000","last_fullsized":"2024-03-04T16:20:47.413835+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_clean_scrub_stamp":"2024-03-04T16:19:30.702539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"240'15","reported_seq":573,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425201+0000","last_change":"2024-03-04T16:16:53.856464+0000","last_active":"2024-03-04T16:20:48.425201+0000","last_peered":"2024-03-04T16:20:48.425201+0000","last_clean":"2024-03-04T16:20:48.425201+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:20:48.425201+0000","last_undegraded":"2024-03-04T16:20:48.425201+0000","last_fullsized":"2024-03-04T16:20:48.425201+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:16:53.856361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:16:53.856361+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.19","version":"390'11860","reported_seq":22283,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425240+0000","last_change":"2024-03-04T16:20:39.141273+0000","last_active":"2024-03-04T16:20:48.425240+0000","last_peered":"2024-03-04T16:20:48.425240+0000","last_clean":"2024-03-04T16:20:48.425240+0000","last_became_active":"2024-03-04T16:20:22.474329+0000","last_became_peered":"2024-03-04T16:20:22.474329+0000","last_unstale":"2024-03-04T16:20:48.425240+0000","last_undegraded":"2024-03-04T16:20:48.425240+0000","last_fullsized":"2024-03-04T16:20:48.425240+0000","mapping_epoch":377,"log_start":"328'8564","ondisk_log_start":"328'8564","created":320,"last_epoch_clean":378,"parent":"0.0","parent_split_bits":5,"last_scrub":"385'11600","last_scrub_stamp":"2024-03-04T16:20:39.140519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:39.140519+0000","objects_scrubbed":860,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:56.761113+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":716,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414353+0000","last_change":"2024-03-04T16:20:07.498249+0000","last_active":"2024-03-04T16:20:47.414353+0000","last_peered":"2024-03-04T16:20:47.414353+0000","last_clean":"2024-03-04T16:20:47.414353+0000","last_became_active":"2024-03-04T16:20:07.497725+0000","last_became_peered":"2024-03-04T16:20:07.497725+0000","last_unstale":"2024-03-04T16:20:47.414353+0000","last_undegraded":"2024-03-04T16:20:47.414353+0000","last_fullsized":"2024-03-04T16:20:47.414353+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:56.474881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:16:56.474881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":699,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415609+0000","last_change":"2024-03-04T16:15:56.777453+0000","last_active":"2024-03-04T16:20:47.415609+0000","last_peered":"2024-03-04T16:20:47.415609+0000","last_clean":"2024-03-04T16:20:47.415609+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:20:47.415609+0000","last_undegraded":"2024-03-04T16:20:47.415609+0000","last_fullsized":"2024-03-04T16:20:47.415609+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:15:56.777347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:15:56.777347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":616,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426716+0000","last_change":"2024-03-04T16:16:51.831340+0000","last_active":"2024-03-04T16:20:48.426716+0000","last_peered":"2024-03-04T16:20:48.426716+0000","last_clean":"2024-03-04T16:20:48.426716+0000","last_became_active":"2024-03-04T16:10:29.242546+0000","last_became_peered":"2024-03-04T16:10:29.242546+0000","last_unstale":"2024-03-04T16:20:48.426716+0000","last_undegraded":"2024-03-04T16:20:48.426716+0000","last_fullsized":"2024-03-04T16:20:48.426716+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:51.830496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:16:51.830496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"240'15","reported_seq":719,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428356+0000","last_change":"2024-03-04T16:18:01.357560+0000","last_active":"2024-03-04T16:20:48.428356+0000","last_peered":"2024-03-04T16:20:48.428356+0000","last_clean":"2024-03-04T16:20:48.428356+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:20:48.428356+0000","last_undegraded":"2024-03-04T16:20:48.428356+0000","last_fullsized":"2024-03-04T16:20:48.428356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:18:01.357413+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:18:01.357413+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"389'11764","reported_seq":12021,"reported_epoch":389,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-04T16:20:48.045394+0000","last_change":"2024-03-04T16:20:39.019838+0000","last_active":"2024-03-04T16:20:48.045394+0000","last_peered":"2024-03-04T16:20:48.045394+0000","last_clean":"2024-03-04T16:20:06.265762+0000","last_became_active":"2024-03-04T16:20:08.765625+0000","last_became_peered":"2024-03-04T16:20:08.765625+0000","last_unstale":"2024-03-04T16:20:48.045394+0000","last_undegraded":"2024-03-04T16:20:48.045394+0000","last_fullsized":"2024-03-04T16:20:48.045394+0000","mapping_epoch":365,"log_start":"324'7649","ondisk_log_start":"324'7649","created":296,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'9547","last_scrub_stamp":"2024-03-04T16:19:21.986416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:19:21.986416+0000","objects_scrubbed":2778,"log_size":4115,"log_dups_size":0,"ondisk_log_size":4115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":622,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36569088,"num_objects":558,"num_object_clones":0,"num_object_copies":1674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":558,"num_objects_unfound":0,"num_objects_dirty":558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,4],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":558}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":716,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415265+0000","last_change":"2024-03-04T16:19:04.784783+0000","last_active":"2024-03-04T16:20:47.415265+0000","last_peered":"2024-03-04T16:20:47.415265+0000","last_clean":"2024-03-04T16:20:47.415265+0000","last_became_active":"2024-03-04T16:19:04.784559+0000","last_became_peered":"2024-03-04T16:19:04.784559+0000","last_unstale":"2024-03-04T16:20:47.415265+0000","last_undegraded":"2024-03-04T16:20:47.415265+0000","last_fullsized":"2024-03-04T16:20:47.415265+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.446216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.446216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":765,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414694+0000","last_change":"2024-03-04T16:20:07.518944+0000","last_active":"2024-03-04T16:20:47.414694+0000","last_peered":"2024-03-04T16:20:47.414694+0000","last_clean":"2024-03-04T16:20:47.414694+0000","last_became_active":"2024-03-04T16:20:07.493115+0000","last_became_peered":"2024-03-04T16:20:07.493115+0000","last_unstale":"2024-03-04T16:20:47.414694+0000","last_undegraded":"2024-03-04T16:20:47.414694+0000","last_fullsized":"2024-03-04T16:20:47.414694+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:17:07.251155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:07.251155+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"240'27","reported_seq":799,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413699+0000","last_change":"2024-03-04T16:20:07.513057+0000","last_active":"2024-03-04T16:20:47.413699+0000","last_peered":"2024-03-04T16:20:47.413699+0000","last_clean":"2024-03-04T16:20:47.413699+0000","last_became_active":"2024-03-04T16:20:07.468416+0000","last_became_peered":"2024-03-04T16:20:07.468416+0000","last_unstale":"2024-03-04T16:20:47.413699+0000","last_undegraded":"2024-03-04T16:20:47.413699+0000","last_fullsized":"2024-03-04T16:20:47.413699+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:17:54.552272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:54.552272+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"366'10","reported_seq":962,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414149+0000","last_change":"2024-03-04T16:20:07.496063+0000","last_active":"2024-03-04T16:20:47.414149+0000","last_peered":"2024-03-04T16:20:47.414149+0000","last_clean":"2024-03-04T16:20:47.414149+0000","last_became_active":"2024-03-04T16:20:07.468442+0000","last_became_peered":"2024-03-04T16:20:07.468442+0000","last_unstale":"2024-03-04T16:20:47.414149+0000","last_undegraded":"2024-03-04T16:20:47.414149+0000","last_fullsized":"2024-03-04T16:20:47.414149+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'9","last_scrub_stamp":"2024-03-04T16:19:34.559612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:19:34.559612+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:59.107226+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"390'11614","reported_seq":16985,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.513198+0000","last_change":"2024-03-04T16:20:35.246114+0000","last_active":"2024-03-04T16:20:48.513198+0000","last_peered":"2024-03-04T16:20:48.513198+0000","last_clean":"2024-03-04T16:20:48.513198+0000","last_became_active":"2024-03-04T16:20:08.569755+0000","last_became_peered":"2024-03-04T16:20:08.569755+0000","last_unstale":"2024-03-04T16:20:48.513198+0000","last_undegraded":"2024-03-04T16:20:48.513198+0000","last_fullsized":"2024-03-04T16:20:48.513198+0000","mapping_epoch":365,"log_start":"326'8105","ondisk_log_start":"326'8105","created":296,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'11219","last_scrub_stamp":"2024-03-04T16:20:35.245770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:35.245770+0000","objects_scrubbed":981,"log_size":3509,"log_dups_size":0,"ondisk_log_size":3509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:50.346988+0000","scrub_duration":399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010880,"num_objects":580,"num_object_clones":0,"num_object_copies":1740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":401,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":691,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423422+0000","last_change":"2024-03-04T16:16:18.274771+0000","last_active":"2024-03-04T16:20:48.423422+0000","last_peered":"2024-03-04T16:20:48.423422+0000","last_clean":"2024-03-04T16:20:48.423422+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:20:48.423422+0000","last_undegraded":"2024-03-04T16:20:48.423422+0000","last_fullsized":"2024-03-04T16:20:48.423422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:18.274659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:18.274659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":731,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414732+0000","last_change":"2024-03-04T16:20:07.497958+0000","last_active":"2024-03-04T16:20:47.414732+0000","last_peered":"2024-03-04T16:20:47.414732+0000","last_clean":"2024-03-04T16:20:47.414732+0000","last_became_active":"2024-03-04T16:20:07.497627+0000","last_became_peered":"2024-03-04T16:20:07.497627+0000","last_unstale":"2024-03-04T16:20:47.414732+0000","last_undegraded":"2024-03-04T16:20:47.414732+0000","last_fullsized":"2024-03-04T16:20:47.414732+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:15.612623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:19:15.612623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"240'18","reported_seq":499,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425371+0000","last_change":"2024-03-04T16:20:43.851864+0000","last_active":"2024-03-04T16:20:48.425371+0000","last_peered":"2024-03-04T16:20:48.425371+0000","last_clean":"2024-03-04T16:20:48.425371+0000","last_became_active":"2024-03-04T16:20:07.488523+0000","last_became_peered":"2024-03-04T16:20:07.488523+0000","last_unstale":"2024-03-04T16:20:48.425371+0000","last_undegraded":"2024-03-04T16:20:48.425371+0000","last_fullsized":"2024-03-04T16:20:48.425371+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:20:43.851736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:20:43.851736+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:22:07.995809+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":113,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.412924+0000","last_change":"2024-03-04T16:19:04.768572+0000","last_active":"2024-03-04T16:20:47.412924+0000","last_peered":"2024-03-04T16:20:47.412924+0000","last_clean":"2024-03-04T16:20:47.412924+0000","last_became_active":"2024-03-04T16:19:04.768239+0000","last_became_peered":"2024-03-04T16:19:04.768239+0000","last_unstale":"2024-03-04T16:20:47.412924+0000","last_undegraded":"2024-03-04T16:20:47.412924+0000","last_fullsized":"2024-03-04T16:20:47.412924+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:16.337706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.337706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"390'11659","reported_seq":14684,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.487371+0000","last_change":"2024-03-04T16:20:23.506903+0000","last_active":"2024-03-04T16:20:48.487371+0000","last_peered":"2024-03-04T16:20:48.487371+0000","last_clean":"2024-03-04T16:20:48.487371+0000","last_became_active":"2024-03-04T16:20:23.480333+0000","last_became_peered":"2024-03-04T16:20:23.480333+0000","last_unstale":"2024-03-04T16:20:48.487371+0000","last_undegraded":"2024-03-04T16:20:48.487371+0000","last_fullsized":"2024-03-04T16:20:48.487371+0000","mapping_epoch":378,"log_start":"327'8202","ondisk_log_start":"327'8202","created":296,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:51.191958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40108032,"num_objects":612,"num_object_clones":0,"num_object_copies":1836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7920,"num_write_kb":273024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1401,"num_bytes_recovered":91815936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":723,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413343+0000","last_change":"2024-03-04T16:19:04.749645+0000","last_active":"2024-03-04T16:20:47.413343+0000","last_peered":"2024-03-04T16:20:47.413343+0000","last_clean":"2024-03-04T16:20:47.413343+0000","last_became_active":"2024-03-04T16:19:04.749223+0000","last_became_peered":"2024-03-04T16:19:04.749223+0000","last_unstale":"2024-03-04T16:20:47.413343+0000","last_undegraded":"2024-03-04T16:20:47.413343+0000","last_fullsized":"2024-03-04T16:20:47.413343+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:57.307587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:57.307587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":113,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.412737+0000","last_change":"2024-03-04T16:19:04.880283+0000","last_active":"2024-03-04T16:20:47.412737+0000","last_peered":"2024-03-04T16:20:47.412737+0000","last_clean":"2024-03-04T16:20:47.412737+0000","last_became_active":"2024-03-04T16:19:04.780445+0000","last_became_peered":"2024-03-04T16:19:04.780445+0000","last_unstale":"2024-03-04T16:20:47.412737+0000","last_undegraded":"2024-03-04T16:20:47.412737+0000","last_fullsized":"2024-03-04T16:20:47.412737+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:15.744737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.744737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":723,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414982+0000","last_change":"2024-03-04T16:17:46.310631+0000","last_active":"2024-03-04T16:20:47.414982+0000","last_peered":"2024-03-04T16:20:47.414982+0000","last_clean":"2024-03-04T16:20:47.414982+0000","last_became_active":"2024-03-04T16:10:29.369980+0000","last_became_peered":"2024-03-04T16:10:29.369980+0000","last_unstale":"2024-03-04T16:20:47.414982+0000","last_undegraded":"2024-03-04T16:20:47.414982+0000","last_fullsized":"2024-03-04T16:20:47.414982+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.310358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.310358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"248'36","reported_seq":525,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426316+0000","last_change":"2024-03-04T16:17:58.315576+0000","last_active":"2024-03-04T16:20:48.426316+0000","last_peered":"2024-03-04T16:20:48.426316+0000","last_clean":"2024-03-04T16:20:48.426316+0000","last_became_active":"2024-03-04T16:10:29.353404+0000","last_became_peered":"2024-03-04T16:10:29.353404+0000","last_unstale":"2024-03-04T16:20:48.426316+0000","last_undegraded":"2024-03-04T16:20:48.426316+0000","last_fullsized":"2024-03-04T16:20:48.426316+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'36","last_scrub_stamp":"2024-03-04T16:17:58.315405+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.315405+0000","objects_scrubbed":9,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":44,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"389'11769","reported_seq":16051,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.751192+0000","last_change":"2024-03-04T16:20:11.582989+0000","last_active":"2024-03-04T16:20:47.751192+0000","last_peered":"2024-03-04T16:20:47.751192+0000","last_clean":"2024-03-04T16:20:47.751192+0000","last_became_active":"2024-03-04T16:20:11.549134+0000","last_became_peered":"2024-03-04T16:20:11.549134+0000","last_unstale":"2024-03-04T16:20:47.751192+0000","last_undegraded":"2024-03-04T16:20:47.751192+0000","last_fullsized":"2024-03-04T16:20:47.751192+0000","mapping_epoch":368,"log_start":"318'7287","ondisk_log_start":"318'7287","created":296,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:49.305475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38961152,"num_objects":594,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7988,"num_write_kb":274656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2026,"num_bytes_recovered":132775936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":598,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413425+0000","last_change":"2024-03-04T16:19:04.779974+0000","last_active":"2024-03-04T16:20:47.413425+0000","last_peered":"2024-03-04T16:20:47.413425+0000","last_clean":"2024-03-04T16:20:47.413425+0000","last_became_active":"2024-03-04T16:19:04.779570+0000","last_became_peered":"2024-03-04T16:19:04.779570+0000","last_unstale":"2024-03-04T16:20:47.413425+0000","last_undegraded":"2024-03-04T16:20:47.413425+0000","last_fullsized":"2024-03-04T16:20:47.413425+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:16.573055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.573055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":119,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414153+0000","last_change":"2024-03-04T16:20:07.487587+0000","last_active":"2024-03-04T16:20:47.414153+0000","last_peered":"2024-03-04T16:20:47.414153+0000","last_clean":"2024-03-04T16:20:47.414153+0000","last_became_active":"2024-03-04T16:20:07.487240+0000","last_became_peered":"2024-03-04T16:20:07.487240+0000","last_unstale":"2024-03-04T16:20:47.414153+0000","last_undegraded":"2024-03-04T16:20:47.414153+0000","last_fullsized":"2024-03-04T16:20:47.414153+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:52.337129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:17:52.337129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":722,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415373+0000","last_change":"2024-03-04T16:18:11.409686+0000","last_active":"2024-03-04T16:20:47.415373+0000","last_peered":"2024-03-04T16:20:47.415373+0000","last_clean":"2024-03-04T16:20:47.415373+0000","last_became_active":"2024-03-04T16:10:29.354592+0000","last_became_peered":"2024-03-04T16:10:29.354592+0000","last_unstale":"2024-03-04T16:20:47.415373+0000","last_undegraded":"2024-03-04T16:20:47.415373+0000","last_fullsized":"2024-03-04T16:20:47.415373+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:11.409565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:18:11.409565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"240'19","reported_seq":589,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424212+0000","last_change":"2024-03-04T16:20:07.538989+0000","last_active":"2024-03-04T16:20:48.424212+0000","last_peered":"2024-03-04T16:20:48.424212+0000","last_clean":"2024-03-04T16:20:48.424212+0000","last_became_active":"2024-03-04T16:20:07.507991+0000","last_became_peered":"2024-03-04T16:20:07.507991+0000","last_unstale":"2024-03-04T16:20:48.424212+0000","last_undegraded":"2024-03-04T16:20:48.424212+0000","last_fullsized":"2024-03-04T16:20:48.424212+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:16:49.762239+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:16:49.762239+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"389'15738","reported_seq":19477,"reported_epoch":389,"state":"peering","last_fresh":"2024-03-04T16:20:48.206957+0000","last_change":"2024-03-04T16:20:48.435020+0000","last_active":"2024-03-04T16:20:48.435020+0000","last_peered":"2024-03-04T16:20:48.206957+0000","last_clean":"2024-03-04T16:20:48.206957+0000","last_became_active":"2024-03-04T16:19:05.473830+0000","last_became_peered":"2024-03-04T16:19:05.473830+0000","last_unstale":"2024-03-04T16:20:48.206957+0000","last_undegraded":"2024-03-04T16:20:48.206957+0000","last_fullsized":"2024-03-04T16:20:48.206957+0000","mapping_epoch":390,"log_start":"353'12303","ondisk_log_start":"353'12303","created":296,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15738,"num_write_kb":538752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1758,"num_bytes_recovered":115146752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":697,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423344+0000","last_change":"2024-03-04T16:20:07.474327+0000","last_active":"2024-03-04T16:20:48.423344+0000","last_peered":"2024-03-04T16:20:48.423344+0000","last_clean":"2024-03-04T16:20:48.423344+0000","last_became_active":"2024-03-04T16:20:07.473964+0000","last_became_peered":"2024-03-04T16:20:07.473964+0000","last_unstale":"2024-03-04T16:20:48.423344+0000","last_undegraded":"2024-03-04T16:20:48.423344+0000","last_fullsized":"2024-03-04T16:20:48.423344+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:05.382402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:05.382402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":729,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415056+0000","last_change":"2024-03-04T16:15:51.894739+0000","last_active":"2024-03-04T16:20:47.415056+0000","last_peered":"2024-03-04T16:20:47.415056+0000","last_clean":"2024-03-04T16:20:47.415056+0000","last_became_active":"2024-03-04T16:10:29.342255+0000","last_became_peered":"2024-03-04T16:10:29.342255+0000","last_unstale":"2024-03-04T16:20:47.415056+0000","last_undegraded":"2024-03-04T16:20:47.415056+0000","last_fullsized":"2024-03-04T16:20:47.415056+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:15:51.894452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:15:51.894452+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":371,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428406+0000","last_change":"2024-03-04T16:20:07.502096+0000","last_active":"2024-03-04T16:20:48.428406+0000","last_peered":"2024-03-04T16:20:48.428406+0000","last_clean":"2024-03-04T16:20:48.428406+0000","last_became_active":"2024-03-04T16:20:07.495218+0000","last_became_peered":"2024-03-04T16:20:07.495218+0000","last_unstale":"2024-03-04T16:20:48.428406+0000","last_undegraded":"2024-03-04T16:20:48.428406+0000","last_fullsized":"2024-03-04T16:20:48.428406+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:18:08.651791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"338'43","reported_seq":1083,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.427946+0000","last_change":"2024-03-04T16:19:04.909779+0000","last_active":"2024-03-04T16:20:48.427946+0000","last_peered":"2024-03-04T16:20:48.427946+0000","last_clean":"2024-03-04T16:20:48.427946+0000","last_became_active":"2024-03-04T16:19:04.769086+0000","last_became_peered":"2024-03-04T16:19:04.769086+0000","last_unstale":"2024-03-04T16:20:48.427946+0000","last_undegraded":"2024-03-04T16:20:48.427946+0000","last_fullsized":"2024-03-04T16:20:48.427946+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'42","last_scrub_stamp":"2024-03-04T16:18:14.457588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.457588+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":561,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.429067+0000","last_change":"2024-03-04T16:20:07.504303+0000","last_active":"2024-03-04T16:20:48.429067+0000","last_peered":"2024-03-04T16:20:48.429067+0000","last_clean":"2024-03-04T16:20:48.429067+0000","last_became_active":"2024-03-04T16:20:07.500834+0000","last_became_peered":"2024-03-04T16:20:07.500834+0000","last_unstale":"2024-03-04T16:20:48.429067+0000","last_undegraded":"2024-03-04T16:20:48.429067+0000","last_fullsized":"2024-03-04T16:20:48.429067+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:01.832115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.832115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"389'11768","reported_seq":17977,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.900456+0000","last_change":"2024-03-04T16:20:34.605087+0000","last_active":"2024-03-04T16:20:47.900456+0000","last_peered":"2024-03-04T16:20:47.900456+0000","last_clean":"2024-03-04T16:20:47.900456+0000","last_became_active":"2024-03-04T16:19:22.395019+0000","last_became_peered":"2024-03-04T16:19:22.395019+0000","last_unstale":"2024-03-04T16:20:47.900456+0000","last_undegraded":"2024-03-04T16:20:47.900456+0000","last_fullsized":"2024-03-04T16:20:47.900456+0000","mapping_epoch":348,"log_start":"324'7620","ondisk_log_start":"324'7620","created":296,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'11362","last_scrub_stamp":"2024-03-04T16:20:34.604838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:34.604838+0000","objects_scrubbed":982,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:52.605566+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":717,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415035+0000","last_change":"2024-03-04T16:19:04.756842+0000","last_active":"2024-03-04T16:20:47.415035+0000","last_peered":"2024-03-04T16:20:47.415035+0000","last_clean":"2024-03-04T16:20:47.415035+0000","last_became_active":"2024-03-04T16:19:04.756529+0000","last_became_peered":"2024-03-04T16:19:04.756529+0000","last_unstale":"2024-03-04T16:20:47.415035+0000","last_undegraded":"2024-03-04T16:20:47.415035+0000","last_fullsized":"2024-03-04T16:20:47.415035+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:50.390204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:50.390204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":101,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":738,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425638+0000","last_change":"2024-03-04T16:19:04.796532+0000","last_active":"2024-03-04T16:20:48.425638+0000","last_peered":"2024-03-04T16:20:48.425638+0000","last_clean":"2024-03-04T16:20:48.425638+0000","last_became_active":"2024-03-04T16:19:04.796249+0000","last_became_peered":"2024-03-04T16:19:04.796249+0000","last_unstale":"2024-03-04T16:20:48.425638+0000","last_undegraded":"2024-03-04T16:20:48.425638+0000","last_fullsized":"2024-03-04T16:20:48.425638+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:17.450946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:17.450946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":121,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.412530+0000","last_change":"2024-03-04T16:20:36.130868+0000","last_active":"2024-03-04T16:20:47.412530+0000","last_peered":"2024-03-04T16:20:47.412530+0000","last_clean":"2024-03-04T16:20:47.412530+0000","last_became_active":"2024-03-04T16:19:04.780845+0000","last_became_peered":"2024-03-04T16:19:04.780845+0000","last_unstale":"2024-03-04T16:20:47.412530+0000","last_undegraded":"2024-03-04T16:20:47.412530+0000","last_fullsized":"2024-03-04T16:20:47.412530+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:36.130570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:20:36.130570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:47.974746+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"240'22","reported_seq":354,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.427393+0000","last_change":"2024-03-04T16:16:57.389891+0000","last_active":"2024-03-04T16:20:48.427393+0000","last_peered":"2024-03-04T16:20:48.427393+0000","last_clean":"2024-03-04T16:20:48.427393+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:20:48.427393+0000","last_undegraded":"2024-03-04T16:20:48.427393+0000","last_fullsized":"2024-03-04T16:20:48.427393+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'22","last_scrub_stamp":"2024-03-04T16:16:57.389758+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:16:57.389758+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":113,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415390+0000","last_change":"2024-03-04T16:19:04.756381+0000","last_active":"2024-03-04T16:20:47.415390+0000","last_peered":"2024-03-04T16:20:47.415390+0000","last_clean":"2024-03-04T16:20:47.415390+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:20:47.415390+0000","last_undegraded":"2024-03-04T16:20:47.415390+0000","last_fullsized":"2024-03-04T16:20:47.415390+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:19.696369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:19.696369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"390'15649","reported_seq":20945,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.546170+0000","last_change":"2024-03-04T16:20:25.424406+0000","last_active":"2024-03-04T16:20:48.546170+0000","last_peered":"2024-03-04T16:20:48.546170+0000","last_clean":"2024-03-04T16:20:48.546170+0000","last_became_active":"2024-03-04T16:20:25.395900+0000","last_became_peered":"2024-03-04T16:20:25.395900+0000","last_unstale":"2024-03-04T16:20:48.546170+0000","last_undegraded":"2024-03-04T16:20:48.546170+0000","last_fullsized":"2024-03-04T16:20:48.546170+0000","mapping_epoch":380,"log_start":"354'12416","ondisk_log_start":"354'12416","created":296,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3233,"log_dups_size":0,"ondisk_log_size":3233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68354048,"num_objects":1043,"num_object_clones":0,"num_object_copies":3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15649,"num_write_kb":534144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2540,"num_bytes_recovered":166461440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":710,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415762+0000","last_change":"2024-03-04T16:20:07.500992+0000","last_active":"2024-03-04T16:20:47.415762+0000","last_peered":"2024-03-04T16:20:47.415762+0000","last_clean":"2024-03-04T16:20:47.415762+0000","last_became_active":"2024-03-04T16:20:07.500640+0000","last_became_peered":"2024-03-04T16:20:07.500640+0000","last_unstale":"2024-03-04T16:20:47.415762+0000","last_undegraded":"2024-03-04T16:20:47.415762+0000","last_fullsized":"2024-03-04T16:20:47.415762+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:46.756659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:46.756659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1025,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.413554+0000","last_change":"2024-03-04T16:19:04.548298+0000","last_active":"2024-03-04T16:20:47.413554+0000","last_peered":"2024-03-04T16:20:47.413554+0000","last_clean":"2024-03-04T16:20:47.413554+0000","last_became_active":"2024-03-04T16:10:32.281095+0000","last_became_peered":"2024-03-04T16:10:32.281095+0000","last_unstale":"2024-03-04T16:20:47.413554+0000","last_undegraded":"2024-03-04T16:20:47.413554+0000","last_fullsized":"2024-03-04T16:20:47.413554+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:19:04.548119+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:19:04.548119+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":715,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423582+0000","last_change":"2024-03-04T16:20:07.492247+0000","last_active":"2024-03-04T16:20:48.423582+0000","last_peered":"2024-03-04T16:20:48.423582+0000","last_clean":"2024-03-04T16:20:48.423582+0000","last_became_active":"2024-03-04T16:20:07.491353+0000","last_became_peered":"2024-03-04T16:20:07.491353+0000","last_unstale":"2024-03-04T16:20:48.423582+0000","last_undegraded":"2024-03-04T16:20:48.423582+0000","last_fullsized":"2024-03-04T16:20:48.423582+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:20.104643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:16:20.104643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"366'30","reported_seq":295,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414667+0000","last_change":"2024-03-04T16:20:07.534827+0000","last_active":"2024-03-04T16:20:47.414667+0000","last_peered":"2024-03-04T16:20:47.414667+0000","last_clean":"2024-03-04T16:20:47.414667+0000","last_became_active":"2024-03-04T16:20:07.498725+0000","last_became_peered":"2024-03-04T16:20:07.498725+0000","last_unstale":"2024-03-04T16:20:47.414667+0000","last_undegraded":"2024-03-04T16:20:47.414667+0000","last_fullsized":"2024-03-04T16:20:47.414667+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'29","last_scrub_stamp":"2024-03-04T16:17:05.303531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.303531+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":729,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414480+0000","last_change":"2024-03-04T16:19:04.750893+0000","last_active":"2024-03-04T16:20:47.414480+0000","last_peered":"2024-03-04T16:20:47.414480+0000","last_clean":"2024-03-04T16:20:47.414480+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:20:47.414480+0000","last_undegraded":"2024-03-04T16:20:47.414480+0000","last_fullsized":"2024-03-04T16:20:47.414480+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:13.429937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:18:13.429937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"389'11902","reported_seq":15631,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.831146+0000","last_change":"2024-03-04T16:20:07.534106+0000","last_active":"2024-03-04T16:20:47.831146+0000","last_peered":"2024-03-04T16:20:47.831146+0000","last_clean":"2024-03-04T16:20:47.831146+0000","last_became_active":"2024-03-04T16:20:07.486061+0000","last_became_peered":"2024-03-04T16:20:07.486061+0000","last_unstale":"2024-03-04T16:20:47.831146+0000","last_undegraded":"2024-03-04T16:20:47.831146+0000","last_fullsized":"2024-03-04T16:20:47.831146+0000","mapping_epoch":364,"log_start":"317'6800","ondisk_log_start":"317'6800","created":296,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:20:49.305450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36012032,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8079,"num_write_kb":276128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1755,"num_bytes_recovered":115015680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":707,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.427510+0000","last_change":"2024-03-04T16:16:55.443627+0000","last_active":"2024-03-04T16:20:48.427510+0000","last_peered":"2024-03-04T16:20:48.427510+0000","last_clean":"2024-03-04T16:20:48.427510+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:20:48.427510+0000","last_undegraded":"2024-03-04T16:20:48.427510+0000","last_fullsized":"2024-03-04T16:20:48.427510+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:55.443547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:16:55.443547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":569,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424023+0000","last_change":"2024-03-04T16:20:07.498875+0000","last_active":"2024-03-04T16:20:48.424023+0000","last_peered":"2024-03-04T16:20:48.424023+0000","last_clean":"2024-03-04T16:20:48.424023+0000","last_became_active":"2024-03-04T16:20:07.498576+0000","last_became_peered":"2024-03-04T16:20:07.498576+0000","last_unstale":"2024-03-04T16:20:48.424023+0000","last_undegraded":"2024-03-04T16:20:48.424023+0000","last_fullsized":"2024-03-04T16:20:48.424023+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:02.856124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:02.856124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":1816,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.674687+0000","last_change":"2024-03-04T16:17:58.263035+0000","last_active":"2024-03-04T16:20:47.674687+0000","last_peered":"2024-03-04T16:20:47.674687+0000","last_clean":"2024-03-04T16:20:47.674687+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:20:47.674687+0000","last_undegraded":"2024-03-04T16:20:47.674687+0000","last_fullsized":"2024-03-04T16:20:47.674687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:17:58.262905+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.262905+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1077,"num_read_kb":1072,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"240'24","reported_seq":627,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424885+0000","last_change":"2024-03-04T16:19:04.893033+0000","last_active":"2024-03-04T16:20:48.424885+0000","last_peered":"2024-03-04T16:20:48.424885+0000","last_clean":"2024-03-04T16:20:48.424885+0000","last_became_active":"2024-03-04T16:19:04.826819+0000","last_became_peered":"2024-03-04T16:19:04.826819+0000","last_unstale":"2024-03-04T16:20:48.424885+0000","last_undegraded":"2024-03-04T16:20:48.424885+0000","last_fullsized":"2024-03-04T16:20:48.424885+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:53.588494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.588494+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":716,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.416149+0000","last_change":"2024-03-04T16:17:43.601364+0000","last_active":"2024-03-04T16:20:47.416149+0000","last_peered":"2024-03-04T16:20:47.416149+0000","last_clean":"2024-03-04T16:20:47.416149+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:20:47.416149+0000","last_undegraded":"2024-03-04T16:20:47.416149+0000","last_fullsized":"2024-03-04T16:20:47.416149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:43.600397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:43.600397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"389'15795","reported_seq":18589,"reported_epoch":389,"state":"peering","last_fresh":"2024-03-04T16:20:48.024386+0000","last_change":"2024-03-04T16:20:48.428180+0000","last_active":"2024-03-04T16:20:48.428180+0000","last_peered":"2024-03-04T16:20:48.024386+0000","last_clean":"2024-03-04T16:20:48.024386+0000","last_became_active":"2024-03-04T16:20:08.645373+0000","last_became_peered":"2024-03-04T16:20:08.645373+0000","last_unstale":"2024-03-04T16:20:48.024386+0000","last_undegraded":"2024-03-04T16:20:48.024386+0000","last_fullsized":"2024-03-04T16:20:48.024386+0000","mapping_epoch":390,"log_start":"353'12369","ondisk_log_start":"353'12369","created":296,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3426,"log_dups_size":0,"ondisk_log_size":3426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73138200,"num_objects":1117,"num_object_clones":0,"num_object_copies":3351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1117,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15795,"num_write_kb":541121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1288,"num_bytes_recovered":84279320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":725,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425594+0000","last_change":"2024-03-04T16:19:04.770031+0000","last_active":"2024-03-04T16:20:48.425594+0000","last_peered":"2024-03-04T16:20:48.425594+0000","last_clean":"2024-03-04T16:20:48.425594+0000","last_became_active":"2024-03-04T16:19:04.769649+0000","last_became_peered":"2024-03-04T16:19:04.769649+0000","last_unstale":"2024-03-04T16:20:48.425594+0000","last_undegraded":"2024-03-04T16:20:48.425594+0000","last_fullsized":"2024-03-04T16:20:48.425594+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_clean_scrub_stamp":"2024-03-04T16:18:09.376470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":723,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.427818+0000","last_change":"2024-03-04T16:19:04.769538+0000","last_active":"2024-03-04T16:20:48.427818+0000","last_peered":"2024-03-04T16:20:48.427818+0000","last_clean":"2024-03-04T16:20:48.427818+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:20:48.427818+0000","last_undegraded":"2024-03-04T16:20:48.427818+0000","last_fullsized":"2024-03-04T16:20:48.427818+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:12.393386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:18:12.393386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"240'37","reported_seq":691,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415398+0000","last_change":"2024-03-04T16:18:09.347836+0000","last_active":"2024-03-04T16:20:47.415398+0000","last_peered":"2024-03-04T16:20:47.415398+0000","last_clean":"2024-03-04T16:20:47.415398+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:20:47.415398+0000","last_undegraded":"2024-03-04T16:20:47.415398+0000","last_fullsized":"2024-03-04T16:20:47.415398+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:18:09.347708+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:18:09.347708+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":426,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415176+0000","last_change":"2024-03-04T16:19:04.784739+0000","last_active":"2024-03-04T16:20:47.415176+0000","last_peered":"2024-03-04T16:20:47.415176+0000","last_clean":"2024-03-04T16:20:47.415176+0000","last_became_active":"2024-03-04T16:19:04.784531+0000","last_became_peered":"2024-03-04T16:19:04.784531+0000","last_unstale":"2024-03-04T16:20:47.415176+0000","last_undegraded":"2024-03-04T16:20:47.415176+0000","last_fullsized":"2024-03-04T16:20:47.415176+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:54.600256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:54.600256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.c","version":"389'15632","reported_seq":21228,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.763376+0000","last_change":"2024-03-04T16:20:24.517314+0000","last_active":"2024-03-04T16:20:47.763376+0000","last_peered":"2024-03-04T16:20:47.763376+0000","last_clean":"2024-03-04T16:20:47.763376+0000","last_became_active":"2024-03-04T16:20:24.498072+0000","last_became_peered":"2024-03-04T16:20:24.498072+0000","last_unstale":"2024-03-04T16:20:47.763376+0000","last_undegraded":"2024-03-04T16:20:47.763376+0000","last_fullsized":"2024-03-04T16:20:47.763376+0000","mapping_epoch":379,"log_start":"351'11831","ondisk_log_start":"351'11831","created":296,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11595","last_scrub_stamp":"2024-03-04T16:19:28.257320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:19:28.257320+0000","objects_scrubbed":5199,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15632,"num_write_kb":535744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2660,"num_bytes_recovered":174325760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":727,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.425387+0000","last_change":"2024-03-04T16:19:04.849240+0000","last_active":"2024-03-04T16:20:48.425387+0000","last_peered":"2024-03-04T16:20:48.425387+0000","last_clean":"2024-03-04T16:20:48.425387+0000","last_became_active":"2024-03-04T16:19:04.834912+0000","last_became_peered":"2024-03-04T16:19:04.834912+0000","last_unstale":"2024-03-04T16:20:48.425387+0000","last_undegraded":"2024-03-04T16:20:48.425387+0000","last_fullsized":"2024-03-04T16:20:48.425387+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:05.755479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:05.755479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":726,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415792+0000","last_change":"2024-03-04T16:19:04.763225+0000","last_active":"2024-03-04T16:20:47.415792+0000","last_peered":"2024-03-04T16:20:47.415792+0000","last_clean":"2024-03-04T16:20:47.415792+0000","last_became_active":"2024-03-04T16:19:04.762915+0000","last_became_peered":"2024-03-04T16:19:04.762915+0000","last_unstale":"2024-03-04T16:20:47.415792+0000","last_undegraded":"2024-03-04T16:20:47.415792+0000","last_fullsized":"2024-03-04T16:20:47.415792+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:01.874356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:37.697324+0000","last_clean_scrub_stamp":"2024-03-04T16:16:01.874356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"240'25","reported_seq":768,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426577+0000","last_change":"2024-03-04T16:20:07.551268+0000","last_active":"2024-03-04T16:20:48.426577+0000","last_peered":"2024-03-04T16:20:48.426577+0000","last_clean":"2024-03-04T16:20:48.426577+0000","last_became_active":"2024-03-04T16:20:07.504270+0000","last_became_peered":"2024-03-04T16:20:07.504270+0000","last_unstale":"2024-03-04T16:20:48.426577+0000","last_undegraded":"2024-03-04T16:20:48.426577+0000","last_fullsized":"2024-03-04T16:20:48.426577+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:17:55.230654+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:17:55.230654+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"195'11","reported_seq":573,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415569+0000","last_change":"2024-03-04T16:16:11.811999+0000","last_active":"2024-03-04T16:20:47.415569+0000","last_peered":"2024-03-04T16:20:47.415569+0000","last_clean":"2024-03-04T16:20:47.415569+0000","last_became_active":"2024-03-04T16:10:29.352294+0000","last_became_peered":"2024-03-04T16:10:29.352294+0000","last_unstale":"2024-03-04T16:20:47.415569+0000","last_undegraded":"2024-03-04T16:20:47.415569+0000","last_fullsized":"2024-03-04T16:20:47.415569+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'11","last_scrub_stamp":"2024-03-04T16:16:11.811878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:16:11.811878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.f","version":"390'15968","reported_seq":23532,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.543136+0000","last_change":"2024-03-04T16:20:16.529900+0000","last_active":"2024-03-04T16:20:48.543136+0000","last_peered":"2024-03-04T16:20:48.543136+0000","last_clean":"2024-03-04T16:20:48.543136+0000","last_became_active":"2024-03-04T16:20:16.411672+0000","last_became_peered":"2024-03-04T16:20:16.411672+0000","last_unstale":"2024-03-04T16:20:48.543136+0000","last_undegraded":"2024-03-04T16:20:48.543136+0000","last_fullsized":"2024-03-04T16:20:48.543136+0000","mapping_epoch":373,"log_start":"353'12510","ondisk_log_start":"353'12510","created":296,"last_epoch_clean":374,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75628544,"num_objects":1154,"num_object_clones":0,"num_object_copies":3462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15968,"num_write_kb":547904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3666,"num_bytes_recovered":240058368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":731,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414153+0000","last_change":"2024-03-04T16:19:21.668870+0000","last_active":"2024-03-04T16:20:47.414153+0000","last_peered":"2024-03-04T16:20:47.414153+0000","last_clean":"2024-03-04T16:20:47.414153+0000","last_became_active":"2024-03-04T16:19:04.758780+0000","last_became_peered":"2024-03-04T16:19:04.758780+0000","last_unstale":"2024-03-04T16:20:47.414153+0000","last_undegraded":"2024-03-04T16:20:47.414153+0000","last_fullsized":"2024-03-04T16:20:47.414153+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:21.668734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:19:21.668734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":577,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424998+0000","last_change":"2024-03-04T16:20:48.211714+0000","last_active":"2024-03-04T16:20:48.424998+0000","last_peered":"2024-03-04T16:20:48.424998+0000","last_clean":"2024-03-04T16:20:48.424998+0000","last_became_active":"2024-03-04T16:20:07.494182+0000","last_became_peered":"2024-03-04T16:20:07.494182+0000","last_unstale":"2024-03-04T16:20:48.424998+0000","last_undegraded":"2024-03-04T16:20:48.424998+0000","last_fullsized":"2024-03-04T16:20:48.424998+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:20:48.211601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:20:48.211601+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:22:01.745627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"240'35","reported_seq":532,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414642+0000","last_change":"2024-03-04T16:20:07.507884+0000","last_active":"2024-03-04T16:20:47.414642+0000","last_peered":"2024-03-04T16:20:47.414642+0000","last_clean":"2024-03-04T16:20:47.414642+0000","last_became_active":"2024-03-04T16:20:07.483694+0000","last_became_peered":"2024-03-04T16:20:07.483694+0000","last_unstale":"2024-03-04T16:20:47.414642+0000","last_undegraded":"2024-03-04T16:20:47.414642+0000","last_fullsized":"2024-03-04T16:20:47.414642+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'35","last_scrub_stamp":"2024-03-04T16:18:16.671642+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:18:16.671642+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"366'9","reported_seq":931,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422826+0000","last_change":"2024-03-04T16:20:07.527060+0000","last_active":"2024-03-04T16:20:48.422826+0000","last_peered":"2024-03-04T16:20:48.422826+0000","last_clean":"2024-03-04T16:20:48.422826+0000","last_became_active":"2024-03-04T16:20:07.484919+0000","last_became_peered":"2024-03-04T16:20:07.484919+0000","last_unstale":"2024-03-04T16:20:48.422826+0000","last_undegraded":"2024-03-04T16:20:48.422826+0000","last_fullsized":"2024-03-04T16:20:48.422826+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:06.654784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:06.654784+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"389'15950","reported_seq":30996,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.768416+0000","last_change":"2024-03-04T16:20:25.432723+0000","last_active":"2024-03-04T16:20:47.768416+0000","last_peered":"2024-03-04T16:20:47.768416+0000","last_clean":"2024-03-04T16:20:47.768416+0000","last_became_active":"2024-03-04T16:20:25.384697+0000","last_became_peered":"2024-03-04T16:20:25.384697+0000","last_unstale":"2024-03-04T16:20:47.768416+0000","last_undegraded":"2024-03-04T16:20:47.768416+0000","last_fullsized":"2024-03-04T16:20:47.768416+0000","mapping_epoch":380,"log_start":"342'11186","ondisk_log_start":"342'11186","created":296,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74055680,"num_objects":1130,"num_object_clones":0,"num_object_copies":3390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15950,"num_write_kb":546560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7407,"num_bytes_recovered":485425152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":665,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423849+0000","last_change":"2024-03-04T16:20:07.491021+0000","last_active":"2024-03-04T16:20:48.423849+0000","last_peered":"2024-03-04T16:20:48.423849+0000","last_clean":"2024-03-04T16:20:48.423849+0000","last_became_active":"2024-03-04T16:20:07.480286+0000","last_became_peered":"2024-03-04T16:20:07.480286+0000","last_unstale":"2024-03-04T16:20:48.423849+0000","last_undegraded":"2024-03-04T16:20:48.423849+0000","last_fullsized":"2024-03-04T16:20:48.423849+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:15.665346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:15.665346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":715,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428395+0000","last_change":"2024-03-04T16:17:47.139524+0000","last_active":"2024-03-04T16:20:48.428395+0000","last_peered":"2024-03-04T16:20:48.428395+0000","last_clean":"2024-03-04T16:20:48.428395+0000","last_became_active":"2024-03-04T16:07:36.221304+0000","last_became_peered":"2024-03-04T16:07:36.221304+0000","last_unstale":"2024-03-04T16:20:48.428395+0000","last_undegraded":"2024-03-04T16:20:48.428395+0000","last_fullsized":"2024-03-04T16:20:48.428395+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:47.139403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:17:47.139403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"240'30","reported_seq":806,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422739+0000","last_change":"2024-03-04T16:18:07.422343+0000","last_active":"2024-03-04T16:20:48.422739+0000","last_peered":"2024-03-04T16:20:48.422739+0000","last_clean":"2024-03-04T16:20:48.422739+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:20:48.422739+0000","last_undegraded":"2024-03-04T16:20:48.422739+0000","last_fullsized":"2024-03-04T16:20:48.422739+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'30","last_scrub_stamp":"2024-03-04T16:18:07.422222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:18:07.422222+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":62,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422471+0000","last_change":"2024-03-04T16:20:07.487523+0000","last_active":"2024-03-04T16:20:48.422471+0000","last_peered":"2024-03-04T16:20:48.422471+0000","last_clean":"2024-03-04T16:20:48.422471+0000","last_became_active":"2024-03-04T16:20:07.487202+0000","last_became_peered":"2024-03-04T16:20:07.487202+0000","last_unstale":"2024-03-04T16:20:48.422471+0000","last_undegraded":"2024-03-04T16:20:48.422471+0000","last_fullsized":"2024-03-04T16:20:48.422471+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:45.302787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:45.302787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"389'11580","reported_seq":14623,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.751407+0000","last_change":"2024-03-04T16:20:37.649489+0000","last_active":"2024-03-04T16:20:47.751407+0000","last_peered":"2024-03-04T16:20:47.751407+0000","last_clean":"2024-03-04T16:20:47.751407+0000","last_became_active":"2024-03-04T16:20:20.449236+0000","last_became_peered":"2024-03-04T16:20:20.449236+0000","last_unstale":"2024-03-04T16:20:47.751407+0000","last_undegraded":"2024-03-04T16:20:47.751407+0000","last_fullsized":"2024-03-04T16:20:47.751407+0000","mapping_epoch":375,"log_start":"324'7731","ondisk_log_start":"324'7731","created":296,"last_epoch_clean":376,"parent":"0.0","parent_split_bits":0,"last_scrub":"385'11283","last_scrub_stamp":"2024-03-04T16:20:37.649151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:37.649151+0000","objects_scrubbed":878,"log_size":3849,"log_dups_size":0,"ondisk_log_size":3849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:41.846062+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37552128,"num_objects":573,"num_object_clones":0,"num_object_copies":1719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":305,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":707,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415526+0000","last_change":"2024-03-04T16:17:53.653243+0000","last_active":"2024-03-04T16:20:47.415526+0000","last_peered":"2024-03-04T16:20:47.415526+0000","last_clean":"2024-03-04T16:20:47.415526+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:20:47.415526+0000","last_undegraded":"2024-03-04T16:20:47.415526+0000","last_fullsized":"2024-03-04T16:20:47.415526+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:53.653131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:17:53.653131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"366'9","reported_seq":97,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415931+0000","last_change":"2024-03-04T16:20:07.500315+0000","last_active":"2024-03-04T16:20:47.415931+0000","last_peered":"2024-03-04T16:20:47.415931+0000","last_clean":"2024-03-04T16:20:47.415931+0000","last_became_active":"2024-03-04T16:20:07.484334+0000","last_became_peered":"2024-03-04T16:20:07.484334+0000","last_unstale":"2024-03-04T16:20:47.415931+0000","last_undegraded":"2024-03-04T16:20:47.415931+0000","last_fullsized":"2024-03-04T16:20:47.415931+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'8","last_scrub_stamp":"2024-03-04T16:17:58.494829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:58.494829+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":711,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.422605+0000","last_change":"2024-03-04T16:17:44.202256+0000","last_active":"2024-03-04T16:20:48.422605+0000","last_peered":"2024-03-04T16:20:48.422605+0000","last_clean":"2024-03-04T16:20:48.422605+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:20:48.422605+0000","last_undegraded":"2024-03-04T16:20:48.422605+0000","last_fullsized":"2024-03-04T16:20:48.422605+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:44.202137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.202137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"301'34","reported_seq":779,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.427715+0000","last_change":"2024-03-04T16:20:07.552791+0000","last_active":"2024-03-04T16:20:48.427715+0000","last_peered":"2024-03-04T16:20:48.427715+0000","last_clean":"2024-03-04T16:20:48.427715+0000","last_became_active":"2024-03-04T16:20:07.507258+0000","last_became_peered":"2024-03-04T16:20:07.507258+0000","last_unstale":"2024-03-04T16:20:48.427715+0000","last_undegraded":"2024-03-04T16:20:48.427715+0000","last_fullsized":"2024-03-04T16:20:48.427715+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'34","last_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:17:05.452417+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"390'11751","reported_seq":14634,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.510968+0000","last_change":"2024-03-04T16:20:40.162786+0000","last_active":"2024-03-04T16:20:48.510968+0000","last_peered":"2024-03-04T16:20:48.510968+0000","last_clean":"2024-03-04T16:20:48.510968+0000","last_became_active":"2024-03-04T16:20:26.399469+0000","last_became_peered":"2024-03-04T16:20:26.399469+0000","last_unstale":"2024-03-04T16:20:48.510968+0000","last_undegraded":"2024-03-04T16:20:48.510968+0000","last_fullsized":"2024-03-04T16:20:48.510968+0000","mapping_epoch":381,"log_start":"328'8449","ondisk_log_start":"328'8449","created":296,"last_epoch_clean":382,"parent":"0.0","parent_split_bits":0,"last_scrub":"385'11527","last_scrub_stamp":"2024-03-04T16:20:40.162316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:40.162316+0000","objects_scrubbed":815,"log_size":3302,"log_dups_size":0,"ondisk_log_size":3302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:49.263192+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":588,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":701,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423195+0000","last_change":"2024-03-04T16:19:04.549243+0000","last_active":"2024-03-04T16:20:48.423195+0000","last_peered":"2024-03-04T16:20:48.423195+0000","last_clean":"2024-03-04T16:20:48.423195+0000","last_became_active":"2024-03-04T16:10:29.418978+0000","last_became_peered":"2024-03-04T16:10:29.418978+0000","last_unstale":"2024-03-04T16:20:48.423195+0000","last_undegraded":"2024-03-04T16:20:48.423195+0000","last_fullsized":"2024-03-04T16:20:48.423195+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:19:04.549160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:19:04.549160+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":725,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.416057+0000","last_change":"2024-03-04T16:20:35.417694+0000","last_active":"2024-03-04T16:20:47.416057+0000","last_peered":"2024-03-04T16:20:47.416057+0000","last_clean":"2024-03-04T16:20:47.416057+0000","last_became_active":"2024-03-04T16:20:07.478023+0000","last_became_peered":"2024-03-04T16:20:07.478023+0000","last_unstale":"2024-03-04T16:20:47.416057+0000","last_undegraded":"2024-03-04T16:20:47.416057+0000","last_fullsized":"2024-03-04T16:20:47.416057+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:35.417616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:20:35.417616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:53.691535+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":669,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.412621+0000","last_change":"2024-03-04T16:20:35.098517+0000","last_active":"2024-03-04T16:20:47.412621+0000","last_peered":"2024-03-04T16:20:47.412621+0000","last_clean":"2024-03-04T16:20:47.412621+0000","last_became_active":"2024-03-04T16:20:07.471591+0000","last_became_peered":"2024-03-04T16:20:07.471591+0000","last_unstale":"2024-03-04T16:20:47.412621+0000","last_undegraded":"2024-03-04T16:20:47.412621+0000","last_fullsized":"2024-03-04T16:20:47.412621+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:35.098310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:20:35.098310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:36.088277+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"240'24","reported_seq":513,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.431090+0000","last_change":"2024-03-04T16:20:07.513149+0000","last_active":"2024-03-04T16:20:48.431090+0000","last_peered":"2024-03-04T16:20:48.431090+0000","last_clean":"2024-03-04T16:20:48.431090+0000","last_became_active":"2024-03-04T16:20:07.471083+0000","last_became_peered":"2024-03-04T16:20:07.471083+0000","last_unstale":"2024-03-04T16:20:48.431090+0000","last_undegraded":"2024-03-04T16:20:48.431090+0000","last_fullsized":"2024-03-04T16:20:48.431090+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:17:00.348771+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:17:00.348771+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3","version":"389'11625","reported_seq":13264,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.789937+0000","last_change":"2024-03-04T16:20:47.458137+0000","last_active":"2024-03-04T16:20:47.789937+0000","last_peered":"2024-03-04T16:20:47.789937+0000","last_clean":"2024-03-04T16:20:47.789937+0000","last_became_active":"2024-03-04T16:20:47.417654+0000","last_became_peered":"2024-03-04T16:20:47.417654+0000","last_unstale":"2024-03-04T16:20:47.789937+0000","last_undegraded":"2024-03-04T16:20:47.789937+0000","last_fullsized":"2024-03-04T16:20:47.789937+0000","mapping_epoch":388,"log_start":"326'8147","ondisk_log_start":"326'8147","created":296,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:16:52.636711+0000","objects_scrubbed":0,"log_size":3478,"log_dups_size":0,"ondisk_log_size":3478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35160064,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":269600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":702,"num_bytes_recovered":46006272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":119,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.415373+0000","last_change":"2024-03-04T16:20:07.503053+0000","last_active":"2024-03-04T16:20:47.415373+0000","last_peered":"2024-03-04T16:20:47.415373+0000","last_clean":"2024-03-04T16:20:47.415373+0000","last_became_active":"2024-03-04T16:20:07.502698+0000","last_became_peered":"2024-03-04T16:20:07.502698+0000","last_unstale":"2024-03-04T16:20:47.415373+0000","last_undegraded":"2024-03-04T16:20:47.415373+0000","last_fullsized":"2024-03-04T16:20:47.415373+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:16:16.301767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:16:16.301767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":717,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.428039+0000","last_change":"2024-03-04T16:20:07.489598+0000","last_active":"2024-03-04T16:20:48.428039+0000","last_peered":"2024-03-04T16:20:48.428039+0000","last_clean":"2024-03-04T16:20:48.428039+0000","last_became_active":"2024-03-04T16:20:07.489274+0000","last_became_peered":"2024-03-04T16:20:07.489274+0000","last_unstale":"2024-03-04T16:20:48.428039+0000","last_undegraded":"2024-03-04T16:20:48.428039+0000","last_fullsized":"2024-03-04T16:20:48.428039+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:17:59.342270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:17:59.342270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.426417+0000","last_change":"2024-03-04T16:20:44.875565+0000","last_active":"2024-03-04T16:20:48.426417+0000","last_peered":"2024-03-04T16:20:48.426417+0000","last_clean":"2024-03-04T16:20:48.426417+0000","last_became_active":"2024-03-04T16:20:07.503387+0000","last_became_peered":"2024-03-04T16:20:07.503387+0000","last_unstale":"2024-03-04T16:20:48.426417+0000","last_undegraded":"2024-03-04T16:20:48.426417+0000","last_fullsized":"2024-03-04T16:20:48.426417+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:44.875393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:20:44.875393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:53.688050+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"240'31","reported_seq":774,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.412871+0000","last_change":"2024-03-04T16:20:07.549108+0000","last_active":"2024-03-04T16:20:47.412871+0000","last_peered":"2024-03-04T16:20:47.412871+0000","last_clean":"2024-03-04T16:20:47.412871+0000","last_became_active":"2024-03-04T16:20:07.502824+0000","last_became_peered":"2024-03-04T16:20:07.502824+0000","last_unstale":"2024-03-04T16:20:47.412871+0000","last_undegraded":"2024-03-04T16:20:47.412871+0000","last_fullsized":"2024-03-04T16:20:47.412871+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'31","last_scrub_stamp":"2024-03-04T16:17:01.304925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:17:01.304925+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"390'11810","reported_seq":14653,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.523893+0000","last_change":"2024-03-04T16:20:41.117294+0000","last_active":"2024-03-04T16:20:48.523893+0000","last_peered":"2024-03-04T16:20:48.523893+0000","last_clean":"2024-03-04T16:20:48.523893+0000","last_became_active":"2024-03-04T16:20:27.410691+0000","last_became_peered":"2024-03-04T16:20:27.410691+0000","last_unstale":"2024-03-04T16:20:48.523893+0000","last_undegraded":"2024-03-04T16:20:48.523893+0000","last_fullsized":"2024-03-04T16:20:48.523893+0000","mapping_epoch":382,"log_start":"328'8522","ondisk_log_start":"328'8522","created":296,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"385'11607","last_scrub_stamp":"2024-03-04T16:20:41.116787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:52.636711+0000","last_clean_scrub_stamp":"2024-03-04T16:20:41.116787+0000","objects_scrubbed":794,"log_size":3288,"log_dups_size":0,"ondisk_log_size":3288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:21:57.322111+0000","scrub_duration":330,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":205,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":659,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.430816+0000","last_change":"2024-03-04T16:20:07.480583+0000","last_active":"2024-03-04T16:20:48.430816+0000","last_peered":"2024-03-04T16:20:48.430816+0000","last_clean":"2024-03-04T16:20:48.430816+0000","last_became_active":"2024-03-04T16:20:07.480213+0000","last_became_peered":"2024-03-04T16:20:07.480213+0000","last_unstale":"2024-03-04T16:20:48.430816+0000","last_undegraded":"2024-03-04T16:20:48.430816+0000","last_fullsized":"2024-03-04T16:20:48.430816+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:11.633381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:18:11.633381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":138,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-03-04T16:20:47.414834+0000","last_change":"2024-03-04T16:19:04.817708+0000","last_active":"2024-03-04T16:20:47.414834+0000","last_peered":"2024-03-04T16:20:47.414834+0000","last_clean":"2024-03-04T16:20:47.414834+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:20:47.414834+0000","last_undegraded":"2024-03-04T16:20:47.414834+0000","last_fullsized":"2024-03-04T16:20:47.414834+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'10","last_scrub_stamp":"2024-03-04T16:17:44.249367+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:17:44.249367+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":721,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.423171+0000","last_change":"2024-03-04T16:20:07.488626+0000","last_active":"2024-03-04T16:20:48.423171+0000","last_peered":"2024-03-04T16:20:48.423171+0000","last_clean":"2024-03-04T16:20:48.423171+0000","last_became_active":"2024-03-04T16:20:07.488309+0000","last_became_peered":"2024-03-04T16:20:07.488309+0000","last_unstale":"2024-03-04T16:20:48.423171+0000","last_undegraded":"2024-03-04T16:20:48.423171+0000","last_fullsized":"2024-03-04T16:20:48.423171+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:18:08.455830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:18:08.455830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"318'25","reported_seq":678,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-03-04T16:20:48.424852+0000","last_change":"2024-03-04T16:20:07.528341+0000","last_active":"2024-03-04T16:20:48.424852+0000","last_peered":"2024-03-04T16:20:48.424852+0000","last_clean":"2024-03-04T16:20:48.424852+0000","last_became_active":"2024-03-04T16:20:07.490099+0000","last_became_peered":"2024-03-04T16:20:07.490099+0000","last_unstale":"2024-03-04T16:20:48.424852+0000","last_undegraded":"2024-03-04T16:20:48.424852+0000","last_fullsized":"2024-03-04T16:20:48.424852+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'25","last_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:18:14.650758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":1180237848,"num_objects":18010,"num_object_clones":0,"num_object_copies":54030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2065,"num_objects_unfound":0,"num_objects_dirty":18010,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":157834,"num_write_kb":5161217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35692,"num_bytes_recovered":2338717720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4974788608,"data_stored":4974772320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96608,"ondisk_log_size":96608,"up":78,"acting":78,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1077,"num_read_kb":1072,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":3802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1112,"num_read_kb":869,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":7322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":475136,"data_stored":12608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":756,"ondisk_log_size":756,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070685,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1307804,"kb_used_data":532492,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92437348,"statfs":{"total":95995035648,"available":94655844352,"internally_reserved":0,"allocated":545271808,"data_stored":544689208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":793901514},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365025,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1546812,"kb_used_data":586864,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92198340,"statfs":{"total":95995035648,"available":94411100160,"internally_reserved":0,"allocated":600948736,"data_stored":600335598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1676204,"kb_used_data":868956,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92068948,"statfs":{"total":95995035648,"available":94278602752,"internally_reserved":0,"allocated":889810944,"data_stored":889212210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411573,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1029092,"kb_used_data":520348,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92716060,"statfs":{"total":95995035648,"available":94941245440,"internally_reserved":0,"allocated":532836352,"data_stored":532293687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,0,0,1,0,8,7],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149936,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1924084,"kb_used_data":886376,"kb_used_omap":1,"kb_used_meta":1037694,"kb_avail":91821068,"statfs":{"total":95995035648,"available":94024773632,"internally_reserved":0,"allocated":907649024,"data_stored":907044054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1062599112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1,2,4,8],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626663,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1452168,"kb_used_data":504696,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92292984,"statfs":{"total":95995035648,"available":94508015616,"internally_reserved":0,"allocated":516808704,"data_stored":516244475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":970193350},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":880873472,"data_stored":880869400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":599064576,"data_stored":599064576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":539033600,"data_stored":539033600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":906366976,"data_stored":906362904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1008537600,"data_stored":1008533528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":500563968,"data_stored":500563968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":540348416,"data_stored":540344344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:20:58.235 INFO:tasks.thrashosds.thrasher:Setting 4.b to [0, 5, 1] 2024-03-04T16:20:58.235 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.b', '0', '5', '1'] 2024-03-04T16:20:58.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pg-upmap 4.b 0 5 1 2024-03-04T16:20:58.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:58.334 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:58.434 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:20:58.437 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:58.463 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:58.565 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:58.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:58.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:58.756 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:58.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:58.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:58 smithi038 ceph-mon[29779]: pgmap v842: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 552 B/s rd, 0 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:58.878 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:58 smithi038 ceph-mon[38113]: pgmap v842: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 552 B/s rd, 0 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:58.886 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:58.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:58.966 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:58 smithi176 ceph-mon[33619]: pgmap v842: 158 pgs: 1 active+clean+scrubbing+deep, 154 active+clean, 3 active+remapped+backfill_toofull; 860 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 552 B/s rd, 0 op/s; 1569/41868 objects misplaced (3.747%) 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178682 10.16 10:688de60d:::benchmark_data_smithi038_155391_object42725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:20:58.967 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 103.550511693, 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96282650700000005, 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178682 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:58.968 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414380+0000", 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2559999999999999e-06 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.969 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414385+0000", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0080000000000002e-06 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414395+0000", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.961e-06 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414402+0000", 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3019999999999997e-06 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.970 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414455+0000", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2580999999999997e-05 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414460+0000", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4310000000000002e-06 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371209+0000", 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95674911100000004 2024-03-04T16:20:58.971 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371328+0000", 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118875 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371653+0000", 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032488900000000003 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.972 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377063+0000", 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054099769999999998 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377128+0000", 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4801000000000005e-05 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377138+0000", 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.006e-05 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.973 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377178+0000", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9690000000000001e-05 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377205+0000", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7565000000000001e-05 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178701 10.16 10:6aa3b8c6:::benchmark_data_smithi038_155391_object42744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:20:58.974 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 103.531101377, 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94355677400000004, 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178701 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.975 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433791+0000", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.294e-06 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433798+0000", 2024-03-04T16:20:58.976 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7100000000000001e-06 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433813+0000", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5096000000000001e-05 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433819+0000", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2659999999999998e-06 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433886+0000", 2024-03-04T16:20:58.977 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7197000000000001e-05 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433895+0000", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3100000000000001e-06 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.373812+0000", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93991698599999995 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.373926+0000", 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011376599999999999 2024-03-04T16:20:58.978 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.374218+0000", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029243200000000002 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377099+0000", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028806830000000002 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377255+0000", 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015678600000000001 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.979 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377284+0000", 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8600999999999999e-05 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377320+0000", 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5969000000000001e-05 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.980 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377346+0000", 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5678e-05 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.981 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178708 10.16 10:6808437f:::benchmark_data_smithi038_155391_object42751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 103.5280305, 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94346813900000004, 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178708 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:58.982 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.983 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436862+0000", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0260000000000002e-06 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436868+0000", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2219999999999999e-06 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436884+0000", 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5888000000000001e-05 2024-03-04T16:20:58.984 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436892+0000", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3750000000000003e-06 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436957+0000", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4307999999999995e-05 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436968+0000", 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1086e-05 2024-03-04T16:20:58.985 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375734+0000", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93876637100000004 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375815+0000", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0813e-05 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375991+0000", 2024-03-04T16:20:58.986 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017593200000000001 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.378898+0000", 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029075699999999999 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.987 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380128+0000", 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00122941 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380235+0000", 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107013 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380295+0000", 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0449999999999999e-05 2024-03-04T16:20:58.988 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380328+0000", 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2675000000000001e-05 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183512 10.9 10:94a4b12e:::benchmark_data_smithi038_155391_object47555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.582389011999993, 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.022642394, 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:58.989 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183512 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:58.990 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382503+0000", 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.962e-06 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382508+0000", 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2329999999999997e-06 2024-03-04T16:20:58.991 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382521+0000", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2875e-05 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382529+0000", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8199999999999997e-06 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382589+0000", 2024-03-04T16:20:58.992 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9803999999999999e-05 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382698+0000", 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108675 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,4,7", 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.383077+0000", 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037943100000000001 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:58.993 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386465+0000", 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033882270000000002 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386614+0000", 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014898 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386666+0000", 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2080999999999998e-05 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.994 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.390710+0000", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040436980000000001 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.390725+0000", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4959e-05 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405084+0000", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0143585209999999 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:58.995 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405144+0000", 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0127999999999998e-05 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183547 10.9 10:96435141:::benchmark_data_smithi038_155391_object47590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.532830863000001, 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97993219899999995, 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:58.996 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183547 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.997 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432061+0000", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432068+0000", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2330000000000003e-06 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432108+0000", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9860000000000001e-05 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:58.998 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432116+0000", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2120000000000003e-06 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432173+0000", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7655000000000003e-05 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432181+0000", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0029999999999994e-06 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:58.999 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405472+0000", 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97329052000000005 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405592+0000", 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120238 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405948+0000", 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035554 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.000 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411767+0000", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058191359999999999 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411869+0000", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010232499999999999 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411887+0000", 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7493999999999999e-05 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.001 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411941+0000", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4728999999999997e-05 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411992+0000", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0201999999999998e-05 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183589 10.9 10:90edb329:::benchmark_data_smithi038_155391_object47632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:20:59.002 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.488783565999995, 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93606389700000003, 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183589 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.003 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476109+0000", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9230000000000001e-06 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476114+0000", 2024-03-04T16:20:59.004 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3159999999999996e-06 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476124+0000", 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0521e-05 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476132+0000", 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9300000000000003e-06 2024-03-04T16:20:59.005 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476187+0000", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4250999999999999e-05 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476196+0000", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0250000000000008e-06 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407549+0000", 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93135321299999996 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.006 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407654+0000", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105425 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407911+0000", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000256269 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412002+0000", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040916559999999999 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.007 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412051+0000", 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8906000000000002e-05 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412095+0000", 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3649999999999997e-05 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412132+0000", 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7221999999999997e-05 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.008 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412171+0000", 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8590000000000002e-05 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183607 10.9 10:9627eeb7:::benchmark_data_smithi038_155391_object47650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.473554230999994, 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93008667, 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.009 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183607 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.010 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491338+0000", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7990000000000001e-06 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491343+0000", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9649999999999999e-06 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.011 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491352+0000", 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4809999999999995e-06 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491360+0000", 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1240000000000005e-06 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491417+0000", 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6093e-05 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.012 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491425+0000", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2409999999999995e-06 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.409749+0000", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.91832424599999996 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.409898+0000", 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014876099999999999 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.013 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.410135+0000", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023713799999999999 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.414069+0000", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039342309999999998 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415821+0000", 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017515129999999999 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.014 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421320+0000", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054993680000000001 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421377+0000", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6845999999999997e-05 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421423+0000", 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5864e-05 2024-03-04T16:20:59.015 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183622 10.9 10:93d2c9ee:::benchmark_data_smithi038_155391_object47665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 97.459583206999994, 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.916269899, 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.016 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183622 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.017 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505309+0000", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.483e-06 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505313+0000", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8400000000000002e-06 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505325+0000", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1219e-05 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505332+0000", 2024-03-04T16:20:59.018 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.515e-06 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505392+0000", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0238000000000002e-05 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505408+0000", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5650000000000001e-05 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412222+0000", 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90681353600000003 2024-03-04T16:20:59.019 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412356+0000", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013452399999999999 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412626+0000", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026958 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415835+0000", 2024-03-04T16:20:59.020 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032093949999999999 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415886+0000", 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0495999999999999e-05 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421486+0000", 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056001439999999996 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.021 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421542+0000", 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5809e-05 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421577+0000", 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5469999999999998e-05 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.023 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223791 10.6 10:6309caeb:::benchmark_data_smithi038_155391_object87834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 52.456669460999997, 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5336528830000002, 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.024 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223791 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.025 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508222+0000", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-06 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508227+0000", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5580000000000002e-06 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508235+0000", 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.79e-06 2024-03-04T16:20:59.026 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508240+0000", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2850000000000002e-06 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508284+0000", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4307000000000001e-05 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508301+0000", 2024-03-04T16:20:59.027 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6427e-05 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.461692+0000", 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95339159900000003 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.461707+0000", 2024-03-04T16:20:59.028 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4426999999999999e-05 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.795533+0000", 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.333826843 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.795668+0000", 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013503899999999999 2024-03-04T16:20:59.029 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.796150+0000", 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00048139299999999999 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.803054+0000", 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006903986 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.030 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.803411+0000", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035698300000000001 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.871742+0000", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.068331431999999998 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041692+0000", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.169949765 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.031 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041759+0000", 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7399999999999998e-05 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041874+0000", 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114273 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254087 10.16 10:6a7c6658:::benchmark_data_smithi038_155391_object118130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:20:59.032 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.49758334, 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99874777000000003, 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254087 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.033 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467309+0000", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.75e-06 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467314+0000", 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1569999999999999e-06 2024-03-04T16:20:59.034 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467325+0000", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1e-05 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467331+0000", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7219999999999996e-06 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467390+0000", 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9556999999999999e-05 2024-03-04T16:20:59.035 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467401+0000", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0740000000000001e-05 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457681+0000", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99027973999999996 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457779+0000", 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8640000000000004e-05 2024-03-04T16:20:59.036 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457885+0000", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105567 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465170+0000", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072846789999999996 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465304+0000", 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013415700000000001 2024-03-04T16:20:59.037 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465960+0000", 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00065629800000000004 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.466018+0000", 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8063000000000003e-05 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.038 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.466055+0000", 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6699999999999998e-05 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254092 10.16 10:69ffd62a:::benchmark_data_smithi038_155391_object118135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:20:59.039 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.491740013999999, 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99644071400000001, 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.040 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254092 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.041 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.042 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473152+0000", 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7549999999999999e-06 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.043 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473157+0000", 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.203e-06 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.044 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473167+0000", 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9590000000000007e-06 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473174+0000", 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0489999999999998e-06 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.045 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473221+0000", 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6257000000000001e-05 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473228+0000", 2024-03-04T16:20:59.046 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5939999999999996e-06 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460640+0000", 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98741175400000003 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.047 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460760+0000", 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120544 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.048 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460882+0000", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121316 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465259+0000", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043770479999999997 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.468940+0000", 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003680853 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.049 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469484+0000", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00054460999999999995 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469539+0000", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4691999999999997e-05 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469591+0000", 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2080000000000003e-05 2024-03-04T16:20:59.050 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254116 10.16 10:6e4c31d2:::benchmark_data_smithi038_155391_object118159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 10.468059292, 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97311181499999999, 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254116 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.051 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.052 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496832+0000", 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3500000000000005e-07 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496836+0000", 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7290000000000002e-06 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496844+0000", 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5709999999999998e-06 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.053 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496849+0000", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9690000000000002e-06 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.498442+0000", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001592996 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.498448+0000", 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.054 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463072+0000", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96462448499999998 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463190+0000", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118097 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463330+0000", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014000300000000001 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.055 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465267+0000", 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019365389999999999 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.468974+0000", 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037075680000000001 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469853+0000", 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000878531 2024-03-04T16:20:59.056 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469902+0000", 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9224999999999997e-05 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469943+0000", 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0676000000000002e-05 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.057 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:255217 10.16 10:6c4c7327:::benchmark_data_smithi038_155391_object119260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:50.543096+0000", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.4217945489999995, 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.089072257000000002, 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 255217 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543096+0000", 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.058 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543096+0000", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543097+0000", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.711e-06 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543103+0000", 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4439999999999999e-06 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.059 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543112+0000", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4609999999999997e-06 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.543121+0000", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2849999999999995e-06 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552344+0000", 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0092232890000000008 2024-03-04T16:20:59.060 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552476+0000", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131994 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552632+0000", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015634500000000001 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.555431+0000", 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002798858 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.061 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.591817+0000", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.036386151999999998 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632090+0000", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.040272529000000001 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632125+0000", 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5295e-05 2024-03-04T16:20:59.062 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632168+0000", 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2894e-05 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:255229 10.6 10:60e51fbb:::benchmark_data_smithi038_155391_object119272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:50.552482+0000", 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 8.4124084099999994, 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.079711324, 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.063 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 255229 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552482+0000", 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.064 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552482+0000", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552484+0000", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8050000000000001e-06 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552492+0000", 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7760000000000003e-06 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.065 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552507+0000", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4824000000000001e-05 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.552513+0000", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8089999999999998e-06 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.554403+0000", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018903139999999999 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.066 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.554544+0000", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000140094 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.554681+0000", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013774600000000001 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.556898+0000", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0022170200000000001 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.557631+0000", 2024-03-04T16:20:59.067 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00073246300000000004 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.558200+0000", 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00056899100000000003 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632087+0000", 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.073887639000000005 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.068 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632129+0000", 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2045000000000001e-05 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:50.632193+0000", 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3798000000000001e-05 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:258226 10.11 10:8a7671fa:::benchmark_data_smithi038_155391_object122269:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:53.654613+0000", 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 5.310277073, 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.077367450000000004, 2024-03-04T16:20:59.069 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 258226 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654613+0000", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654613+0000", 2024-03-04T16:20:59.070 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654615+0000", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.26e-06 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654623+0000", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.131e-06 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654637+0000", 2024-03-04T16:20:59.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.484e-05 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654646+0000", 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.7350000000000002e-06 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654709+0000", 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3027999999999994e-05 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.072 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654809+0000", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9339000000000006e-05 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.654901+0000", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2355999999999995e-05 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.656907+0000", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020063519999999999 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.658536+0000", 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0016289970000000001 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.731872+0000", 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.073335866 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.731933+0000", 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0460000000000001e-05 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:53.731981+0000", 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8086e-05 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259082 10.6 10:66146a1d:::benchmark_data_smithi038_155391_object123125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3815568699999998, 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17653590499999999, 2024-03-04T16:20:59.075 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259082 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583335+0000", 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5039999999999999e-06 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583339+0000", 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3059999999999999e-06 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583348+0000", 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0659999999999993e-06 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583354+0000", 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.378e-06 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583400+0000", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6449999999999997e-05 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583492+0000", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1613999999999994e-05 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583574+0000", 2024-03-04T16:20:59.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2582000000000003e-05 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587766+0000", 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041915850000000003 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587819+0000", 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3044999999999998e-05 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.588257+0000", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043791799999999998 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759743+0000", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.171485729 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759788+0000", 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5698000000000002e-05 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759869+0000", 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1030000000000002e-05 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259084 10.6 10:649bf51c:::benchmark_data_smithi038_155391_object123127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3814196049999996, 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.180419145, 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.082 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259084 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583474+0000", 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3549999999999999e-06 2024-03-04T16:20:59.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583478+0000", 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6260000000000003e-06 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583482+0000", 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6679999999999999e-06 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585467+0000", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019849939999999999 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585595+0000", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012844000000000001 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585735+0000", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013928899999999999 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.588323+0000", 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025887800000000002 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.589825+0000", 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015020369999999999 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763632+0000", 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17380702000000001 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763717+0000", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4268e-05 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763777+0000", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0584000000000001e-05 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763890+0000", 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011249 2024-03-04T16:20:59.089 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259086 10.11 10:8dc5ffb0:::benchmark_data_smithi038_155391_object123129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3805645499999999, 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.179469561, 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259086 2024-03-04T16:20:59.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:20:59.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584327+0000", 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9500000000000001e-07 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584330+0000", 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.117e-06 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584335+0000", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6930000000000004e-06 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584341+0000", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8479999999999997e-06 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584375+0000", 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3337999999999998e-05 2024-03-04T16:20:59.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584457+0000", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2729e-05 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584576+0000", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118279 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.586907+0000", 2024-03-04T16:20:59.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023310290000000001 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587025+0000", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118304 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763696+0000", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17667132799999999 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763753+0000", 2024-03-04T16:20:59.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6672000000000002e-05 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763795+0000", 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2429000000000003e-05 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259089 10.16 10:6e6ec3de:::benchmark_data_smithi038_155391_object123132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:20:59.096 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 4.3778829029999997, 2024-03-04T16:20:59.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17683321099999999, 2024-03-04T16:20:59.097 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.097 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259089 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:20:59.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587009+0000", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9920000000000002e-06 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587015+0000", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587026+0000", 2024-03-04T16:20:59.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1127999999999999e-05 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587033+0000", 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5499999999999997e-06 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587799+0000", 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076608299999999998 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587902+0000", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102632 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587965+0000", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3398999999999994e-05 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.589944+0000", 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019787400000000001 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.590363+0000", 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000418928 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763652+0000", 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17328922399999999 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.102 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763746+0000", 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3706e-05 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763841+0000", 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4510000000000001e-05 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259608 10.16 10:6d95926b:::benchmark_data_smithi038_155391_object123651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:20:59.103 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 3.5538700199999997, 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15055500899999999, 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259608 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411022+0000", 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.637e-06 2024-03-04T16:20:59.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411027+0000", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4940000000000003e-06 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411041+0000", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3390999999999999e-05 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411049+0000", 2024-03-04T16:20:59.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2859999999999999e-06 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411108+0000", 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8727999999999997e-05 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411197+0000", 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9566000000000003e-05 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:20:59.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411272+0000", 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4139000000000003e-05 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.413150+0000", 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018781780000000001 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.413612+0000", 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000462058 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561402+0000", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.147790011 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561525+0000", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012313499999999999 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561575+0000", 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0386000000000002e-05 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.111 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.112 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:20:59.112 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:20:59.112 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:20:59.138 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:20:59.207 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:20:59.240 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:59.254 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:20:59.267 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:59.296 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:20:59.341 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:20:59.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:20:59.381 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:20:59.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:59.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:20:59.426 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:20:59.468 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:20:59.512 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:20:59.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:20:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:20:59.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:20:59.650 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:20:59.679 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:20:59.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:20:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:20:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:20:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:59 smithi176 ceph-mon[33619]: 10.b deep-scrub ok 2024-03-04T16:20:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:59 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:20:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:20:59 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2794145690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:20:59.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:20:59.951 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:20:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:20:59.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:20:59.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:21:00.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[29779]: 10.b deep-scrub ok 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2794145690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[38113]: 10.b deep-scrub ok 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:21:00.075 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:20:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2794145690' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.108 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:00.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:00.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:00.259 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:00.340 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:00.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:00.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.396 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:00.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:00.497 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:00.542 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:00.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:00.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:00.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:00.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:00.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:00.903 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:00.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:00.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: pgmap v843: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 997 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:21:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: pgmap v843: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 997 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:21:00.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:00 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:21:01.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:01.074 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:01.078 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: pgmap v843: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 997 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:21:01.079 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:00 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:21:01.176 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:01.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:01.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:01.350 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:01.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.495 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:01.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:01.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:01.625 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:01.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:01.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:01.852 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:01.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:01.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:01.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:02.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:02.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:02.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:02.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:02.294 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:02.396 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:02.436 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:02.538 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:02.555 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:02.563 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:02.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:21:02.665 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:02.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:02.816 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:02.839 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:02.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:02 smithi176 ceph-mon[33619]: pgmap v844: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:02.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:02.988 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:02.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:02.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:02 smithi038 ceph-mon[29779]: pgmap v844: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:02.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:02 smithi038 ceph-mon[38113]: pgmap v844: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 587 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 1078/28749 objects misplaced (3.750%) 2024-03-04T16:21:03.091 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:03.129 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:03.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.262 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:03.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:03.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:03.406 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:03.507 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:03.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:03.538 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:03.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:03.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:21:03.640 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:03.663 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:03.742 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:03.765 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:03.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:03.881 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:03.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:03.923 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:03.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:04.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:04.061 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-mon[33619]: 4.5 deep-scrub starts 2024-03-04T16:21:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-mon[33619]: 4.5 deep-scrub ok 2024-03-04T16:21:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:21:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:21:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:03 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:04.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:04.190 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:04.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[29779]: 4.5 deep-scrub starts 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[29779]: 4.5 deep-scrub ok 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[38113]: 4.5 deep-scrub starts 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[38113]: 4.5 deep-scrub ok 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:21:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:03 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:04.292 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:04.334 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:04.436 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:04.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:04.567 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:04.607 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:04.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:04.747 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:04.849 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:04.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:04.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:04.976 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:05.003 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:05.061 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:04 smithi176 ceph-mon[33619]: pgmap v845: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 454 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 911/22377 objects misplaced (4.071%) 2024-03-04T16:21:05.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:05.135 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:05.237 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:05.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:04 smithi038 ceph-mon[29779]: pgmap v845: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 454 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 911/22377 objects misplaced (4.071%) 2024-03-04T16:21:05.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:04 smithi038 ceph-mon[38113]: pgmap v845: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 454 MiB data, 6.5 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 911/22377 objects misplaced (4.071%) 2024-03-04T16:21:05.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.279 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:05.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:05.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:05.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:05.526 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:05.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:05.632 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:05.659 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:05.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:05.691 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:05.691 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:05.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.816 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:05.917 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:05.942 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:05.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:05.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:21:05.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:21:05.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:21:05.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:21:05.945 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:05 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:21:06.045 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:06.094 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:06.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:21:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:21:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:21:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:21:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:21:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:05 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:21:06.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:06.221 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:06.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.354 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:06.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:06.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:21:06.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:06.486 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:06.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:06.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:06.724 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:06.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:06.771 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:06.873 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:06.900 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:06.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:06.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:07.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:07.031 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:07.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:07.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:07.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:07.301 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:07.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:07.303 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Removed 135954 objects 2024-03-04T16:21:07.303 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Clean up completed and total clean up time :153.289 2024-03-04T16:21:07.403 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:07.406 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:07.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:07.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:07.552 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:07.653 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.678 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:07.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[29779]: pgmap v846: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 375 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[38113]: pgmap v846: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 375 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:07.705 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:07 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:07.717 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-04T16:21:07.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-04T16:21:07.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:07.802 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:07 smithi176 ceph-mon[33619]: pgmap v846: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 375 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:07 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:07 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:07.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:07.929 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:07.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:08.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:08.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.056 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:08.157 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:08.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:08.283 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:08.307 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:08.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:08.400 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:08.400 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:08.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:08.439 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:08.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:08 smithi176 ceph-mon[33619]: osdmap e392: 8 total, 8 up, 6 in 2024-03-04T16:21:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:08 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:21:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:08 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:21:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:08 smithi176 ceph-mon[33619]: pgmap v848: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:08.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:08.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[29779]: osdmap e392: 8 total, 8 up, 6 in 2024-03-04T16:21:08.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:21:08.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:21:08.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[29779]: pgmap v848: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:08.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:08.624 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[38113]: osdmap e392: 8 total, 8 up, 6 in 2024-03-04T16:21:08.624 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:21:08.624 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:21:08.624 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[38113]: pgmap v848: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 251 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s; 485/12624 objects misplaced (3.842%) 2024-03-04T16:21:08.624 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:08.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:08.692 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:08.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:08.819 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:08.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:08.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:08.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:09.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:21:09.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:09.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.071 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:09.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:09.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:09.173 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:09.197 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:09.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:09.253 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:09.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.324 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:09.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:09.414 INFO:teuthology.orchestra.run.smithi038.stderr:set 4.b pg_upmap mapping to [0,5,1] 2024-03-04T16:21:09.414 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:21:09.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:09.448 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:09.550 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:09.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: osdmap e393: 8 total, 8 up, 6 in 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: 6.1d deep-scrub starts 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: 6.1d deep-scrub ok 2024-03-04T16:21:09.649 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]: dispatch 2024-03-04T16:21:09.676 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:09.688 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: osdmap e393: 8 total, 8 up, 6 in 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: 6.1d deep-scrub starts 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: 6.1d deep-scrub ok 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]: dispatch 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: osdmap e393: 8 total, 8 up, 6 in 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: 6.1d deep-scrub starts 2024-03-04T16:21:09.689 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:21:09.690 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: 6.1d deep-scrub ok 2024-03-04T16:21:09.690 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]: dispatch 2024-03-04T16:21:09.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:09.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:09.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:09.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:09.951 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:09.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:09.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:10.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:10.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.078 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:10.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:10.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:10.206 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:10.309 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:10.333 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:10.435 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:10.462 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:10.563 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:10.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:10.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.718 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:10.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:10.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: 10.d scrub starts 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: 10.d scrub ok 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]': finished 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: osdmap e394: 8 total, 8 up, 6 in 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: pgmap v851: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[29779]: osdmap e395: 8 total, 8 up, 6 in 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: 10.d scrub starts 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: 10.d scrub ok 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]': finished 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: osdmap e394: 8 total, 8 up, 6 in 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: pgmap v851: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:10.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:10 smithi038 ceph-mon[38113]: osdmap e395: 8 total, 8 up, 6 in 2024-03-04T16:21:10.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: 10.d scrub starts 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: 10.d scrub ok 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/816471127' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:21:10.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2446305803' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.b", "id": [0, 5, 1]}]': finished 2024-03-04T16:21:10.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: osdmap e394: 8 total, 8 up, 6 in 2024-03-04T16:21:10.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: pgmap v851: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:10.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:10 smithi176 ceph-mon[33619]: osdmap e395: 8 total, 8 up, 6 in 2024-03-04T16:21:10.847 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:10.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:10.975 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:10.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:11.076 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:11.101 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:11.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:11.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:11.331 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.356 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:11.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:11.385 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:11.385 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:11.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:11.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:11.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:11.608 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:11.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:11.734 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:11.836 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:11.859 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:11.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:11.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:11.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:12.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:21:12.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:12.212 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:12.237 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:12.239 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:12.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:12.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:12.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:12.489 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:12.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:12.614 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:12.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:12.716 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.741 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:12.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:12.841 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:12.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:12.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:12.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:12 smithi038 ceph-mon[29779]: pgmap v853: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:12.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:12 smithi038 ceph-mon[38113]: pgmap v853: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:12.993 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:12.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:13.070 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:12 smithi176 ceph-mon[33619]: pgmap v853: 158 pgs: 155 active+clean, 3 active+remapped+backfill_toofull; 9.5 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s; 220/1026 objects misplaced (21.442%) 2024-03-04T16:21:13.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:13.118 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:13.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:13.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:13.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:13.370 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:13.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:13.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.496 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:13.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:13.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:21:13.563 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:13.597 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:13.607 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:13.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:13.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:13.745 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:13.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:13.870 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:13.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:13.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:13.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[29779]: osdmap e396: 8 total, 8 up, 6 in 2024-03-04T16:21:13.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:13.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:13.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:13.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[38113]: osdmap e396: 8 total, 8 up, 6 in 2024-03-04T16:21:13.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:13 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:13.925 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:13.925 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:13.926 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:13 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:13.926 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:13 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:13.926 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:13 smithi176 ceph-mon[33619]: osdmap e396: 8 total, 8 up, 6 in 2024-03-04T16:21:13.926 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:13 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:13.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:13.996 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:13.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:14.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:14.121 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:14.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:14.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:14.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:14.372 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:14.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:14.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:14.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:14.588 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:21:14.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:14.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:14.693 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:14.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.782 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:14.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:14.831 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:21:14.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:14 smithi176 ceph-mon[33619]: pgmap v855: 158 pgs: 156 active+clean, 2 active+remapped+backfill_toofull; 3.0 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 968 B/s rd, 0 B/s wr, 387 op/s; 116/714 objects misplaced (16.246%) 2024-03-04T16:21:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:14 smithi176 ceph-mon[33619]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:21:14.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:14.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:14.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:14.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:15.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:15.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:15.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:14 smithi038 ceph-mon[29779]: pgmap v855: 158 pgs: 156 active+clean, 2 active+remapped+backfill_toofull; 3.0 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 968 B/s rd, 0 B/s wr, 387 op/s; 116/714 objects misplaced (16.246%) 2024-03-04T16:21:15.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:14 smithi038 ceph-mon[29779]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:21:15.050 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:14 smithi038 ceph-mon[38113]: pgmap v855: 158 pgs: 156 active+clean, 2 active+remapped+backfill_toofull; 3.0 MiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 968 B/s rd, 0 B/s wr, 387 op/s; 116/714 objects misplaced (16.246%) 2024-03-04T16:21:15.050 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:14 smithi038 ceph-mon[38113]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-04T16:21:15.134 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:15.158 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:15.174 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:21:15.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:21:15.212 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:21:15.232 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:21:15.252 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:21:15.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:15.272 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:21:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:15.294 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:21:15.314 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:21:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:21:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:21:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:21:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:21:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:21:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:21:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:21:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:21:15.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:15.388 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_3' removed 2024-03-04T16:21:15.426 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:15.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:15.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:15.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:15.679 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:15.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:15.804 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:15.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:15.945 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:15.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:15.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:21:15.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2583552311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:15.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:21:15.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[29779]: osdmap e397: 8 total, 8 up, 6 in 2024-03-04T16:21:15.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:21:15.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2583552311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:15.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:21:15.984 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:15 smithi038 ceph-mon[38113]: osdmap e397: 8 total, 8 up, 6 in 2024-03-04T16:21:16.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.072 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:16.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:21:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2583552311' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3659346853' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:21:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:15 smithi176 ceph-mon[33619]: osdmap e397: 8 total, 8 up, 6 in 2024-03-04T16:21:16.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:16.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:16.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:16.324 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:16.426 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:16.450 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:16.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:16.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:16.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:16.700 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:16.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:16.826 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:16.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:16.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:16.946 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:16 smithi176 ceph-mon[33619]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:16.946 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:16 smithi176 ceph-mon[33619]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:21:16.952 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:16.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:17.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:17.071 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:21:17.071 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:21:17.071 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:21:17.071 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:21:17.071 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223783 10.1 10:81d9e410:::benchmark_data_smithi038_155391_object87826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.570724776999995, 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5165916839999998, 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223783 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.072 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499136+0000", 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.159e-06 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.073 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499144+0000", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1720000000000001e-06 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499157+0000", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-05 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499168+0000", 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0419e-05 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.074 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499759+0000", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059170200000000005 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499771+0000", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1362999999999999e-05 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457342+0000", 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95757115599999998 2024-03-04T16:21:17.075 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457355+0000", 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2853999999999999e-05 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693012+0000", 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2356567140000001 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.076 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693143+0000", 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131736 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693541+0000", 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039770799999999998 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695555+0000", 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020144350000000002 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.077 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.709538+0000", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013982938 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.781590+0000", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072051207000000006 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015577+0000", 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.233987377 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.078 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015635+0000", 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7565000000000002e-05 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015726+0000", 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1419e-05 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223787 10.10 10:0e9e46b3:::benchmark_data_smithi038_155391_object87830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:17.079 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.565840894999994, 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2362359610000002, 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223787 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.080 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504019+0000", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1850000000000001e-06 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504024+0000", 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.081 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504034+0000", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2569999999999996e-06 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504040+0000", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6619999999999999e-06 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504078+0000", 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8262000000000001e-05 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.082 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504086+0000", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.199e-06 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457609+0000", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95352290900000003 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457620+0000", 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1059e-05 2024-03-04T16:21:17.083 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.575586+0000", 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.11796674 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.575688+0000", 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010215 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,4,5", 2024-03-04T16:21:17.084 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.576009+0000", 2024-03-04T16:21:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000320515 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.577642+0000", 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001632582 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.577832+0000", 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019084700000000001 2024-03-04T16:21:17.086 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.585829+0000", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079967200000000006 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740059+0000", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15422976999999999 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740124+0000", 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5452999999999998e-05 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.087 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740254+0000", 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129715 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223796 10.1 10:811c42ae:::benchmark_data_smithi038_155391_object87839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 70.548509194000005, 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.500384639, 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.088 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223796 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:21:17.089 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521352+0000", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7e-06 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521356+0000", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9089999999999999e-06 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521367+0000", 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0553999999999999e-05 2024-03-04T16:21:17.090 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521375+0000", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9359999999999999e-06 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521424+0000", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9509000000000003e-05 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521432+0000", 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8709999999999997e-06 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.091 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457381+0000", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93594915300000003 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457390+0000", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6470000000000004e-06 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695587+0000", 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.238196968 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.092 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695704+0000", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116767 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695954+0000", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000250344 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.697420+0000", 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014657100000000001 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.093 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.709570+0000", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012149927 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.781633+0000", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072062899999999999 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021608+0000", 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.239974826 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.094 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021646+0000", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8849e-05 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021734+0000", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8069e-05 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226044 10.10 10:0f7e88de:::benchmark_data_smithi038_155391_object90087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:21:17.095 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.524456291999996, 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.015020867, 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226044 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.096 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545405+0000", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545410+0000", 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6520000000000003e-06 2024-03-04T16:21:17.097 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545423+0000", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2283000000000001e-05 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545430+0000", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5959999999999998e-06 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545484+0000", 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4190000000000001e-05 2024-03-04T16:21:17.098 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545495+0000", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0893000000000001e-05 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.553734+0000", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0082391690000001 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.553864+0000", 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129911 2024-03-04T16:21:17.099 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.554229+0000", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036436599999999997 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560197+0000", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059684860000000003 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560250+0000", 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2518000000000001e-05 2024-03-04T16:21:17.100 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560311+0000", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1168999999999997e-05 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560367+0000", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6440999999999997e-05 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560424+0000", 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6212999999999999e-05 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.101 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226056 10.10 10:0bb15e44:::benchmark_data_smithi038_155391_object90099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.505442361999997, 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.003719327, 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226056 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.102 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564418+0000", 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7150000000000001e-06 2024-03-04T16:21:17.103 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564423+0000", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5059999999999997e-06 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564432+0000", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1579999999999994e-06 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564437+0000", 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3709999999999999e-06 2024-03-04T16:21:17.104 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564494+0000", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6125999999999998e-05 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564503+0000", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2720000000000003e-06 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556397+0000", 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99189406999999996 2024-03-04T16:21:17.105 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556524+0000", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677000000000001 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556761+0000", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000237197 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560524+0000", 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037634790000000001 2024-03-04T16:21:17.106 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560573+0000", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8464000000000002e-05 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568028+0000", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074555200000000002 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568086+0000", 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8031000000000001e-05 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.107 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568136+0000", 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9648000000000003e-05 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226060 10.10 10:0d44b17f:::benchmark_data_smithi038_155391_object90103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.502853215000002, 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0032575459999999, 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.108 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226060 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:21:17.109 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567007+0000", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4050000000000001e-06 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567011+0000", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.245e-06 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567021+0000", 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5559999999999996e-06 2024-03-04T16:21:17.110 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567026+0000", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3879999999999999e-06 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567081+0000", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4713999999999998e-05 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567090+0000", 2024-03-04T16:21:17.111 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.755e-06 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558313+0000", 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99122295500000002 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558387+0000", 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4598999999999995e-05 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.112 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558605+0000", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000218035 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560630+0000", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002024292 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566064+0000", 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054337939999999996 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.113 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.114 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570179+0000", 2024-03-04T16:21:17.114 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041149940000000003 2024-03-04T16:21:17.114 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.114 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.114 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.115 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570221+0000", 2024-03-04T16:21:17.115 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2447000000000002e-05 2024-03-04T16:21:17.115 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570263+0000", 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2367000000000003e-05 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226104 10.10 10:0ea17222:::benchmark_data_smithi038_155391_object90147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.615935+0000", 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.453924032000003, 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95449731400000004, 2024-03-04T16:21:17.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226104 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615935+0000", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615935+0000", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615937+0000", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6819999999999999e-06 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615942+0000", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8450000000000002e-06 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615950+0000", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8060000000000001e-06 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.615956+0000", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6409999999999999e-06 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.616013+0000", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6715000000000002e-05 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.616019+0000", 2024-03-04T16:21:17.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9599999999999997e-06 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560667+0000", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94464848400000001 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560781+0000", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000113983 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.561000+0000", 2024-03-04T16:21:17.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00021923600000000001 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566133+0000", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0051327300000000003 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.567910+0000", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017770679999999999 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570312+0000", 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002401837 2024-03-04T16:21:17.121 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570393+0000", 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1298000000000007e-05 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570432+0000", 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9029000000000002e-05 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226117 10.10 10:0e911e1e:::benchmark_data_smithi038_155391_object90160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.122 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.627190+0000", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.442668729000005, 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94342538799999998, 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226117 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627190+0000", 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.123 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627190+0000", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627192+0000", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8390000000000001e-06 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627197+0000", 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2229999999999998e-06 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.124 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627210+0000", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2278e-05 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627218+0000", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0690000000000002e-06 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627293+0000", 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4881000000000004e-05 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.125 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.627303+0000", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.028e-05 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.562526+0000", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93522278000000003 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.562646+0000", 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012048 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.126 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.562881+0000", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023449399999999999 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566186+0000", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033049939999999999 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.567918+0000", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017323079999999999 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.127 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570496+0000", 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025781089999999999 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570576+0000", 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9653000000000005e-05 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570616+0000", 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0000000000000003e-05 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.128 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226158 10.10 10:0ba66ebc:::benchmark_data_smithi038_155391_object90201:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.680207+0000", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.389652244000004, 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.89061830600000003, 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226158 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.129 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680207+0000", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680207+0000", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680208+0000", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.609e-06 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.130 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680216+0000", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.2629999999999997e-06 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680230+0000", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4446e-05 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680236+0000", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8039999999999998e-06 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.131 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680299+0000", 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3099e-05 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.680308+0000", 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9800000000000004e-06 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.564432+0000", 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.88412435599999994 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.132 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.564544+0000", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000111684 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.564722+0000", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017825999999999999 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.567953+0000", 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032312090000000001 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.133 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568148+0000", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019494799999999999 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570657+0000", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025083459999999998 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570736+0000", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9172999999999996e-05 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.134 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570825+0000", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9129000000000007e-05 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226186 10.10 10:0c21c309:::benchmark_data_smithi038_155391_object90229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.709424+0000", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.360434726999998, 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87938501999999996, 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.135 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226186 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709424+0000", 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709424+0000", 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.136 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709426+0000", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.547e-06 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709431+0000", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2170000000000002e-06 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709443+0000", 2024-03-04T16:21:17.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:17.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.222e-05 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709457+0000", 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3635000000000001e-05 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709516+0000", 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9227000000000001e-05 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.139 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.709525+0000", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9129999999999992e-06 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566214+0000", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85668898299999996 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566332+0000", 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011786999999999999 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.140 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566562+0000", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022985900000000001 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570137+0000", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0035750180000000001 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570838+0000", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00070121799999999998 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.141 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588741+0000", 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017902761 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588783+0000", 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1845999999999999e-05 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588809+0000", 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6706000000000001e-05 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.142 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226193 10.10 10:09d78ac9:::benchmark_data_smithi038_155391_object90236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.718537+0000", 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.351321916000003, 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.87036920600000001, 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226193 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.143 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718537+0000", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718537+0000", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718539+0000", 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6330000000000001e-06 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.144 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718544+0000", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0370000000000002e-06 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718558+0000", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4165000000000001e-05 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718566+0000", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1680000000000004e-06 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718622+0000", 2024-03-04T16:21:17.145 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5451000000000002e-05 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.718630+0000", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1589999999999993e-06 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568190+0000", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84956047199999996 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568321+0000", 2024-03-04T16:21:17.146 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013067999999999999 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568542+0000", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00022158900000000001 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570875+0000", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023324840000000001 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570886+0000", 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1416999999999999e-05 2024-03-04T16:21:17.147 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588841+0000", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.017955103 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588880+0000", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8692999999999997e-05 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588906+0000", 2024-03-04T16:21:17.148 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6154999999999999e-05 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226198 10.10 10:0c01a0c9:::benchmark_data_smithi038_155391_object90241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.720101+0000", 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.349758121999997, 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.86888388400000005, 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226198 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.149 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720101+0000", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720101+0000", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720102+0000", 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.097e-06 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.150 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720107+0000", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5140000000000003e-06 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720115+0000", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8289999999999997e-06 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720121+0000", 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4779999999999999e-06 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.151 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720176+0000", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5541000000000002e-05 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.720185+0000", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9469999999999992e-06 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570910+0000", 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85072433599999997 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.152 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.571002+0000", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2479999999999995e-05 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.571348+0000", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00034542999999999998 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.578072+0000", 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0067246390000000001 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.153 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.578110+0000", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7798e-05 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588935+0000", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010824591999999999 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588963+0000", 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8133e-05 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.154 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.588985+0000", 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.207e-05 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226205 10.10 10:08bf4d10:::benchmark_data_smithi038_155391_object90248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.734087+0000", 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.335771541, 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.86039195899999998, 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.155 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226205 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734087+0000", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734087+0000", 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.156 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734090+0000", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0930000000000001e-06 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734094+0000", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5920000000000002e-06 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734103+0000", 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3550000000000005e-06 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.157 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734108+0000", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7219999999999996e-06 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734151+0000", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3087999999999998e-05 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.734157+0000", 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3530000000000002e-06 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.158 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.573108+0000", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.83895129499999999 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.573212+0000", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104102 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.573490+0000", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00027774599999999999 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.578121+0000", 2024-03-04T16:21:17.159 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046308820000000002 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.578157+0000", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6693999999999998e-05 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594397+0000", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.016239637000000001 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594450+0000", 2024-03-04T16:21:17.160 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2537999999999997e-05 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594479+0000", 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9862e-05 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226210 10.10 10:0dd2ccab:::benchmark_data_smithi038_155391_object90253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.740495+0000", 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 64.329363603000004, 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.85413815900000001, 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.161 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226210 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740495+0000", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740495+0000", 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.162 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740498+0000", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.176e-06 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740504+0000", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9900000000000002e-06 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740516+0000", 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2485e-05 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.163 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740524+0000", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9789999999999993e-06 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740566+0000", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2043999999999999e-05 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.740573+0000", 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4139999999999999e-06 2024-03-04T16:21:17.164 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.574956+0000", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.83438360899999997 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.575052+0000", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.577e-05 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.575244+0000", 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000191621 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.165 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.578125+0000", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028818950000000002 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.581547+0000", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034212439999999999 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594531+0000", 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012984124 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.166 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594587+0000", 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6483000000000001e-05 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.594634+0000", 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6325000000000001e-05 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.167 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230566 10.1 10:85b5cd01:::benchmark_data_smithi038_155391_object94609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.634273962999998, 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.039178022, 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230566 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:21:17.168 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435586+0000", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.378e-06 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435591+0000", 2024-03-04T16:21:17.169 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1919999999999998e-06 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435597+0000", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8050000000000001e-06 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435603+0000", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1000000000000003e-06 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435671+0000", 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8094000000000002e-05 2024-03-04T16:21:17.170 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435683+0000", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2036e-05 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465007+0000", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0293241689999999 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465098+0000", 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0810999999999995e-05 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.171 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465450+0000", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035259899999999999 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.470910+0000", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054596419999999998 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.470990+0000", 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0610000000000002e-05 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.172 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474680+0000", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036890199999999999 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474735+0000", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5658999999999997e-05 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474763+0000", 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7906999999999999e-05 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.173 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230589 10.1 10:80d77cf6:::benchmark_data_smithi038_155391_object94632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.598409908000001, 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0184885720000001, 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230589 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.174 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471451+0000", 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9939999999999999e-06 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.175 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471457+0000", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6110000000000001e-06 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471469+0000", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2109999999999999e-05 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471475+0000", 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.331e-06 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.176 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471545+0000", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9733999999999993e-05 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471555+0000", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7999999999999993e-06 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467374+0000", 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99581904700000001 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.177 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467451+0000", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7729000000000001e-05 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467611+0000", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015940400000000001 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472118+0000", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045073149999999996 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.178 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479527+0000", 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074085660000000001 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489845+0000", 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010318539999999999 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489895+0000", 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0170999999999998e-05 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.179 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489938+0000", 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2219999999999999e-05 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230616 10.1 10:858a950a:::benchmark_data_smithi038_155391_object94659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.555751075000003, 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97592274199999995, 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.180 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230616 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.181 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514110+0000", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.652e-06 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514115+0000", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0769999999999998e-06 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514125+0000", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9529999999999994e-06 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.182 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514130+0000", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5269999999999998e-06 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514176+0000", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6238e-05 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514182+0000", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0299999999999999e-06 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.183 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468424+0000", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95424162899999998 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468529+0000", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104589 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468781+0000", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025258799999999999 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472121+0000", 2024-03-04T16:21:17.184 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033400690000000002 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479568+0000", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074465130000000001 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489972+0000", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010404681000000001 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490006+0000", 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3556999999999997e-05 2024-03-04T16:21:17.185 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490031+0000", 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4638999999999999e-05 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230624 10.1 10:8414195d:::benchmark_data_smithi038_155391_object94667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.544132484999999, 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96442188399999995, 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.186 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230624 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.187 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525729+0000", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0169999999999999e-06 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525735+0000", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0939999999999996e-06 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525745+0000", 2024-03-04T16:21:17.188 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0638e-05 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525752+0000", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.199e-06 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525802+0000", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9988000000000003e-05 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525809+0000", 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1700000000000002e-06 2024-03-04T16:21:17.189 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469771+0000", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94396249899999995 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469840+0000", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8458000000000001e-05 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469999+0000", 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000159801 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.190 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474635+0000", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046351999999999999 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479605+0000", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049702929999999998 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490099+0000", 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010494111 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.191 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490126+0000", 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6585999999999999e-05 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490148+0000", 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.283e-05 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230641 10.1 10:80f3e66f:::benchmark_data_smithi038_155391_object94684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:21:17.192 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.550847+0000", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.519012218, 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93937467699999999, 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230641 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550847+0000", 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.193 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550847+0000", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550848+0000", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.361e-06 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550853+0000", 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0390000000000004e-06 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.194 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550863+0000", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0013000000000001e-05 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550871+0000", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9589999999999995e-06 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550926+0000", 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5163e-05 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.195 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.550936+0000", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9860000000000006e-06 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.471021+0000", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92008424300000002 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.471119+0000", 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8486999999999994e-05 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.196 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.471352+0000", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023334199999999999 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.475799+0000", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044466779999999999 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479642+0000", 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0038426089999999999 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.197 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490168+0000", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010526535 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490197+0000", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.9142999999999999e-05 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490221+0000", 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4119000000000001e-05 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.198 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230656 10.1 10:8364ed19:::benchmark_data_smithi038_155391_object94699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.565396+0000", 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 57.504463383000001, 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.92754640799999999, 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230656 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.199 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565396+0000", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565396+0000", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565397+0000", 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.342e-06 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.200 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565402+0000", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5600000000000004e-06 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565412+0000", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0579e-05 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565421+0000", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5199999999999997e-06 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.201 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565467+0000", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6112000000000002e-05 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.565473+0000", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.46e-06 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472151+0000", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90667809399999999 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472230+0000", 2024-03-04T16:21:17.202 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8963999999999998e-05 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472432+0000", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000201988 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.475802+0000", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033698880000000001 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.485248+0000", 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0094460039999999992 2024-03-04T16:21:17.203 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.492845+0000", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0075970270000000001 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.492878+0000", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3006000000000002e-05 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.492942+0000", 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3863999999999997e-05 2024-03-04T16:21:17.204 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.205 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.205 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:21:17.205 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:21:17.205 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:21:17.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:16 smithi038 ceph-mon[29779]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:17.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:16 smithi038 ceph-mon[29779]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:21:17.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:16 smithi038 ceph-mon[38113]: pgmap v857: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:17.205 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:16 smithi038 ceph-mon[38113]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-03-04T16:21:17.239 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.264 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:17.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:17.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:17.376 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:21:17.376 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:21:17.376 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-04T16:21:17.376 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.769254 2024-03-04T16:21:17.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 3 0.7692536864704457 2024-03-04T16:21:17.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:17.491 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.515 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:17.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:17.616 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.640 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:17.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:17.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:17.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:17.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:17.870 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-04T16:21:17.871 INFO:tasks.radosbench:Beginning radosbench... 2024-03-04T16:21:17.871 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-04T16:21:17.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-04T16:21:17.883 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:21:17.890 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:17.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:17.992 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:18.017 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:18.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:18.150 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:18.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:18.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:18.282 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:18.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:18.413 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:18.434 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:18.442 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:18.515 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:18.541 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:18.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:18.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:18.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:18.791 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: osdmap e398: 8 total, 8 up, 6 in 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1317639594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:18.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[29779]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: osdmap e398: 8 total, 8 up, 6 in 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1317639594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:18.823 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:18 smithi038 ceph-mon[38113]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:18.894 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:18.920 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:18.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:19.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:19.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:19.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: pgmap v858: 132 pgs: 132 active+clean; 582 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:21:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: osdmap e398: 8 total, 8 up, 6 in 2024-03-04T16:21:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1317639594' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:18 smithi176 ceph-mon[33619]: from='client.35819 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:19.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:19.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:19.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:19.274 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:19.300 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:19.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:19.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:21:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.428 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:19.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:19.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:19.551 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:19.653 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:19.677 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:19.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:19.807 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:19.909 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:19.932 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:19.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:20.034 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:20.062 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:19 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:19 smithi176 ceph-mon[33619]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:19 smithi176 ceph-mon[33619]: osdmap e399: 8 total, 8 up, 6 in 2024-03-04T16:21:20.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.193 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:20.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:20.226 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:20.226 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[29779]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:20.226 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[29779]: osdmap e399: 8 total, 8 up, 6 in 2024-03-04T16:21:20.227 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:20.227 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[38113]: from='client.35819 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:20.227 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:19 smithi038 ceph-mon[38113]: osdmap e399: 8 total, 8 up, 6 in 2024-03-04T16:21:20.294 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:20.321 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:20.422 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:20.449 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:20.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:20.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:20.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:20.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:20.700 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:20.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:20.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:20.831 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:20.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:20.956 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:20.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:21.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:21.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:21.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:21.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:21.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:21.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:21.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:21.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:21.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:21 smithi038 ceph-mon[29779]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:21.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:21 smithi038 ceph-mon[38113]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:21.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:21.595 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:21.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:21.721 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:21.823 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:21.846 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:21.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:21 smithi176 ceph-mon[33619]: pgmap v861: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:21:21.948 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:21.977 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:21.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:22.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:22.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:22.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.226 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:22.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:22.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:21:22.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:22.355 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:22.457 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:22.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:22.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:22.609 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:22.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:22.734 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:22.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:22 smithi038 ceph-mon[29779]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:21:22.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:22 smithi038 ceph-mon[38113]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:21:22.835 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:22.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:22 smithi176 ceph-mon[33619]: pgmap v862: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:21:22.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:22.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:22.992 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:22.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:23.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:23.121 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:23.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.247 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:23.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:23.348 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:23.376 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:23.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:23.498 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:23.504 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:23.530 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:23.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:23.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:23.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:23.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.625 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:23.626 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:23.631 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:23.732 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:23.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:23.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:23.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:23 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:23.861 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:23.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:23.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:23.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:24.015 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:24.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:24.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:24.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:24.266 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:24.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.393 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:24.429 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 0.769254 (8504132) 2024-03-04T16:21:24.448 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_4' created 2024-03-04T16:21:24.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:24.518 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:24.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:24.643 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]': finished 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: osdmap e400: 8 total, 8 up, 6 in 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2005989497' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='client.35825 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]: dispatch 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]': finished 2024-03-04T16:21:24.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: osdmap e400: 8 total, 8 up, 6 in 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2005989497' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='client.35825 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:21:24.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]: dispatch 2024-03-04T16:21:24.744 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:24.772 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [0, 5, 3, 2]}]': finished 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [1, 2]}]': finished 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 2]}]': finished 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: osdmap e400: 8 total, 8 up, 6 in 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2005989497' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='client.35825 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: pgmap v864: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:21:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]: dispatch 2024-03-04T16:21:24.874 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:24.897 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:24.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:24.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:25.027 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:25.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:25.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-04T16:21:25.154 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:25.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:25.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:25.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.413 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:25.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[29779]: from='client.35825 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-04T16:21:25.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]': finished 2024-03-04T16:21:25.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[29779]: osdmap e401: 8 total, 8 up, 6 in 2024-03-04T16:21:25.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:25.537 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:25.639 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:25.666 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[38113]: from='client.35825 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-04T16:21:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]': finished 2024-03-04T16:21:25.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:25 smithi038 ceph-mon[38113]: osdmap e401: 8 total, 8 up, 6 in 2024-03-04T16:21:25.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:25.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:25 smithi176 ceph-mon[33619]: from='client.35825 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-04T16:21:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3425005187' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 0.7692536864704457}]': finished 2024-03-04T16:21:25.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:25 smithi176 ceph-mon[33619]: osdmap e401: 8 total, 8 up, 6 in 2024-03-04T16:21:25.892 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:25.919 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:25.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:26.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:26.047 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:26.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:26.175 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:26.186 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:26.197 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:21:26.197 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:21:26.197 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 0 2024-03-04T16:21:26.197 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:21:26.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:26.304 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:26.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:26.433 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:26.535 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:26.559 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:26.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.690 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:26.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:26.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:26 smithi038 ceph-mon[29779]: osdmap e402: 8 total, 8 up, 6 in 2024-03-04T16:21:26.733 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:26 smithi038 ceph-mon[29779]: pgmap v867: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:26.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:26 smithi038 ceph-mon[38113]: osdmap e402: 8 total, 8 up, 6 in 2024-03-04T16:21:26.734 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:26 smithi038 ceph-mon[38113]: pgmap v867: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:26.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.815 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:26.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:26.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:26 smithi176 ceph-mon[33619]: osdmap e402: 8 total, 8 up, 6 in 2024-03-04T16:21:26.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:26 smithi176 ceph-mon[33619]: pgmap v867: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:26.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:26.939 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:26.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:27.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:27.066 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:27.086 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:21:27.106 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:21:27.125 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:21:27.145 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:21:27.166 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:21:27.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:27.186 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:21:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:27.207 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:21:27.228 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:21:27.229 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:21:27.293 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:27.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:27.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:27.442 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:27.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:21:27.544 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:27.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:27.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:27.694 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:27.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[29779]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:21:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[29779]: osdmap e403: 8 total, 8 up, 6 in 2024-03-04T16:21:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1932007570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[38113]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:21:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[38113]: osdmap e403: 8 total, 8 up, 6 in 2024-03-04T16:21:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:27 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1932007570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:27.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:27.819 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:27.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:27 smithi176 ceph-mon[33619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-04T16:21:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:27 smithi176 ceph-mon[33619]: osdmap e403: 8 total, 8 up, 6 in 2024-03-04T16:21:27.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:27 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1932007570' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:27.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:27.945 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:27.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:28.047 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:28.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:28.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.201 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:28.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:28.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:28.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:28.427 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:28.455 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:28.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:28.583 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:28.685 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.712 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:28.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:28.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:28 smithi038 ceph-mon[29779]: pgmap v869: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail 2024-03-04T16:21:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:28 smithi038 ceph-mon[38113]: pgmap v869: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail 2024-03-04T16:21:28.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:28.841 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:28.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:28 smithi176 ceph-mon[33619]: pgmap v869: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail 2024-03-04T16:21:28.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:28.972 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:28.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:29.078 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:29.102 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:29.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:29.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:29.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:29.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:29.480 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:29.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:29.605 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:29.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:29.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:29.837 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:29.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:29.962 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:29.990 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:29.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:30.081 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:30.091 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:30.119 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:30.221 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:30.248 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:30.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:30.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:30.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:30.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:30.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:30.634 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:30.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:30.760 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:30.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:30.915 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:30.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:31.017 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:31.040 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:31.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:31.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:31.272 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:31.296 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:31.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:21:31.398 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:31.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:31.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:31.550 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:31.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:31.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:31 smithi038 ceph-mon[29779]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:21:31.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:31 smithi038 ceph-mon[38113]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:21:31.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:31.679 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:31.780 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:31.807 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:31 smithi176 ceph-mon[33619]: pgmap v870: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 341 B/s rd, 0 op/s 2024-03-04T16:21:31.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:31.933 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:31.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:32.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:32.062 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:32.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:32.187 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:32.289 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:32.317 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:32.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:32.444 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:32.546 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:32.573 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:32.668 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:32 smithi038 ceph-mon[29779]: pgmap v871: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 281 B/s rd, 0 op/s 2024-03-04T16:21:32.668 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:32 smithi038 ceph-mon[38113]: pgmap v871: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 281 B/s rd, 0 op/s 2024-03-04T16:21:32.675 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:32.703 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:32.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:32.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.829 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:32.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:32.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:32 smithi176 ceph-mon[33619]: pgmap v871: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 281 B/s rd, 0 op/s 2024-03-04T16:21:32.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:32.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:32.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:33.056 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:33.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:33.185 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:33.212 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:33.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:33.337 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:33.428 INFO:teuthology.orchestra.run.smithi038.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-04T16:21:33.428 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3210586543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[29779]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3210586543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.429 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:33 smithi038 ceph-mon[38113]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:33.442 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:21:33.468 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:33.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.594 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:33.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:33.696 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:33.724 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:33.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-04T16:21:33.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3210586543' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:33 smithi176 ceph-mon[33619]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:21:33.760 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:33.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:33.853 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:33.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:33.978 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:33.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:34.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:34.103 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:34.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:34.228 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:34.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:34.353 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:34.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:34.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:34.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:34.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:34.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:34.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:34.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: osdmap e404: 8 total, 8 up, 6 in 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[38113]: pgmap v873: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: osdmap e404: 8 total, 8 up, 6 in 2024-03-04T16:21:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:34 smithi038 ceph-mon[29779]: pgmap v873: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:34.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:34.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:34.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:21:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1910951070' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-04T16:21:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:21:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: osdmap e404: 8 total, 8 up, 6 in 2024-03-04T16:21:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:34 smithi176 ceph-mon[33619]: pgmap v873: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:34.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:34.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:34.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:34.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:35.101 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:35.125 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:35.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:35.227 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.251 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:35.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:35.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:35.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:35.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:35.502 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:35.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2] out_osds: [6, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:21:35.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:21:35.590 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-04T16:21:35.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd in 6 2024-03-04T16:21:35.605 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:35.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:35.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:35.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[29779]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:21:35.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:21:35.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:21:35.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[29779]: osdmap e405: 8 total, 8 up, 6 in 2024-03-04T16:21:35.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[38113]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:21:35.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:21:35.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:21:35.675 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:35 smithi038 ceph-mon[38113]: osdmap e405: 8 total, 8 up, 6 in 2024-03-04T16:21:35.730 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:35.754 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:35.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:35 smithi176 ceph-mon[33619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-04T16:21:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:35 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:21:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:35 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:21:35.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:35 smithi176 ceph-mon[33619]: osdmap e405: 8 total, 8 up, 6 in 2024-03-04T16:21:35.880 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:35.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:35.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:36.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:36.059 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:21:36.059 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:21:36.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:36.132 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:36.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:36.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:36.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:36.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:36.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:36.511 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:36.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:36.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:36.703 DEBUG:teuthology.orchestra.run.smithi038:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-04T16:21:36.705 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-04T16:21:36.706 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-04T16:21:36.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:36.740 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:36.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:36 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:36.764 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: pgmap v875: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 895 B/s rd, 0 op/s 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:36 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:36.862 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:hints = 1 2024-03-04T16:21:36.862 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-04T16:21:36.863 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object prefix: benchmark_data_smithi038_182448 2024-03-04T16:21:36.863 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:21:36.863 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 0 0 0 0 0 0 - 0 2024-03-04T16:21:36.866 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:36.899 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:36.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:37.001 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.046 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:37.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:37.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:21:37.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:37.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:37.275 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:37.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:37.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:37.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:37.465 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:21:37.513 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:21:37.554 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:37.557 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:21:37.581 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:37.604 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[29779]: osdmap e406: 8 total, 8 up, 6 in 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3678765260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[38113]: osdmap e406: 8 total, 8 up, 6 in 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:21:37.605 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:37 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3678765260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:37.645 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:21:37.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:37.688 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:21:37.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.724 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:37.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:37.732 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:21:37.773 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:21:37.779 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:21:37.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.855 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:37.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:37.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:37 smithi176 ceph-mon[33619]: osdmap e406: 8 total, 8 up, 6 in 2024-03-04T16:21:37.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:37 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:21:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:37 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:21:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:37 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3678765260' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:37.862 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 1 16 2471 2455 153.428 153.438 0.0113338 0.00648177 2024-03-04T16:21:37.956 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:37.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:21:37.982 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:37.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:38.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:38.140 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:38.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:38.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:38.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.424 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:38.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:38.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[29779]: pgmap v877: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:38.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:21:38.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:21:38.454 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:38.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[38113]: pgmap v877: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:38.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:21:38.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:21:38.455 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:38.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:38.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:38.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.707 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:38.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:38.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:38.848 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:38.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:38 smithi176 ceph-mon[33619]: pgmap v877: 148 pgs: 148 active+clean; 582 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:21:38.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:38 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:21:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:38 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:21:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:38.862 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 2 16 4714 4698 146.802 140.188 0.00387696 0.0067963 2024-03-04T16:21:38.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:38.978 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:38.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:39.079 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:39.124 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:39.169 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:39.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:39.227 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:39.293 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:39.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:39.421 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:39.522 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:21:39.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:21:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:21:39.583 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:21:39.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:21:39.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:21:39.584 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:39 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:21:39.585 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:39.687 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:39.713 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:39.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:39.815 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:39.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:39.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:21:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:39 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:21:39.863 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 3 16 6841 6825 142.15 132.938 0.0077236 0.00701546 2024-03-04T16:21:39.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:40.002 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:40.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:40.154 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:40.256 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:40.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:40.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:40.414 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:40.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:40.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:40.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: osdmap e407: 8 total, 8 up, 6 in 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: pgmap v879: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 41 MiB/s wr, 652 op/s 2024-03-04T16:21:40.670 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: osdmap e407: 8 total, 8 up, 6 in 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: pgmap v879: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 41 MiB/s wr, 652 op/s 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:21:40.671 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:21:40.672 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:40 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:21:40.672 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:40.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:40.817 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:40.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:21:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:21:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: osdmap e407: 8 total, 8 up, 6 in 2024-03-04T16:21:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:21:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: pgmap v879: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 41 MiB/s wr, 652 op/s 2024-03-04T16:21:40.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:21:40.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:21:40.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:40 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:21:40.863 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 4 15 9298 9283 145.015 153.625 0.00703412 0.00688687 2024-03-04T16:21:40.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:40.957 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:40.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:41.059 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:41.085 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:41.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:21:41.187 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:41.215 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:41.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.374 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:41.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:41.475 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:41.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:41.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:41.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.633 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:21:41.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:21:41.676 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:41 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:21:41.734 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:41.762 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:21:41.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:41 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:21:41.863 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:41.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 5 16 11552 11536 144.17 140.812 0.00185807 0.00689431 2024-03-04T16:21:41.888 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:41.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:42.019 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:42.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:42.164 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:42.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:42.266 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:42.294 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:42.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:42.424 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:42.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.559 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:42.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:42.660 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:42.688 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:42.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:42.817 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: pgmap v880: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 812 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:21:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:42 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:21:42.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 6 16 13573 13557 141.192 126.312 0.00423445 0.00707187 2024-03-04T16:21:42.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:42.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:42.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:42.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:42.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: pgmap v880: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 812 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: pgmap v880: 148 pgs: 148 active+clean; 245 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 812 B/s rd, 39 MiB/s wr, 621 op/s 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:21:42.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:42 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:21:43.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:43.110 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:43.212 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:43.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:43.365 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:43.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:43.516 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:43.544 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:43.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:21:43.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:21:43.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:21:43.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:21:43.597 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:43.597 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:21:43.597 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:21:43.597 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:21:43.597 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:21:43.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:43.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:43.788 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:43.815 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:43 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:21:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:43 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:21:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:43 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:21:43.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:43 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:21:43.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:43.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 7 16 15410 15394 137.422 114.812 0.0118118 0.00726709 2024-03-04T16:21:43.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:43.943 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:43.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:43.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:43.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:21:43.804+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e407 definitely_dead 0 2024-03-04T16:21:44.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:44.077 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:44.179 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:44.220 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:44.322 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:44.351 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:44.453 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:44.491 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:44.549 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.6. 2024-03-04T16:21:44.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.617 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:44.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:44.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:44.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:44.797 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: pgmap v881: 148 pgs: 148 active+clean; 633 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3511841065' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:21:44.798 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: pgmap v881: 148 pgs: 148 active+clean; 633 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3511841065' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.799 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:44 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:21:44.844 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:44.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:44.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: pgmap v881: 148 pgs: 148 active+clean; 633 MiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3511841065' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: from='client.35870 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-04T16:21:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:44 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:21:44.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 8 16 16740 16724 130.635 83.125 0.0287688 0.00735976 2024-03-04T16:21:44.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:44.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:44.993 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:44.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:45.095 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:45.119 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:45.220 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:45.244 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:45.346 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:45.369 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:45.471 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:45.496 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:45.556 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-04T16:21:45.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:45.621 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:45.724 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:45.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:45.850 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: osdmap e408: 8 total, 8 up, 7 in 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:21:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:21:45.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:45 smithi176 ceph-mon[33619]: osdmap e409: 8 total, 8 up, 7 in 2024-03-04T16:21:45.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 9 16 16740 16724 116.121 0 - 0.00735976 2024-03-04T16:21:45.874 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:45.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: osdmap e408: 8 total, 8 up, 7 in 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:21:45.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[29779]: osdmap e409: 8 total, 8 up, 7 in 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: from='client.35870 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: osdmap e408: 8 total, 8 up, 7 in 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:21:45.898 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:45 smithi038 ceph-mon[38113]: osdmap e409: 8 total, 8 up, 7 in 2024-03-04T16:21:45.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:45.999 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:46.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.125 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:46.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:46.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:46.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:46.351 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:46.375 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:46.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:46.527 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:46.629 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:46.672 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:21:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:21:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[29779]: pgmap v884: 148 pgs: 148 active+clean; 920 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:21:46.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[29779]: osdmap e410: 8 total, 8 up, 7 in 2024-03-04T16:21:46.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:21:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:21:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[38113]: pgmap v884: 148 pgs: 148 active+clean; 920 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:21:46.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:46 smithi038 ceph-mon[38113]: osdmap e410: 8 total, 8 up, 7 in 2024-03-04T16:21:46.774 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:46.805 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:46.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:46 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:21:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:46 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:21:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:46 smithi176 ceph-mon[33619]: pgmap v884: 148 pgs: 148 active+clean; 920 MiB data, 7.5 GiB used, 529 GiB / 536 GiB avail; 819 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-04T16:21:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:46 smithi176 ceph-mon[33619]: osdmap e410: 8 total, 8 up, 7 in 2024-03-04T16:21:46.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 10 16 17282 17266 107.897 16.9375 0.00805165 0.00925953 2024-03-04T16:21:46.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:46.953 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:46.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:47.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:47.069 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:47.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.127 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:47.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:47.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:47.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:47.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:47.418 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:21:47.440 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:47.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:47.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:47.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:47.730 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:47.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:47.832 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:47 smithi176 ceph-mon[33619]: osdmap e411: 8 total, 8 up, 7 in 2024-03-04T16:21:47.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 11 16 18786 18770 106.633 94 0.0147585 0.00936981 2024-03-04T16:21:47.865 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:47.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:47.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:47.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[29779]: osdmap e411: 8 total, 8 up, 7 in 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:21:47.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:21:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:21:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1020648814' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:21:47.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:47 smithi038 ceph-mon[38113]: osdmap e411: 8 total, 8 up, 7 in 2024-03-04T16:21:48.012 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:48.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:48.148 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:48.250 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:48.281 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:48.383 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:48.425 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:48.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:48.561 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:48.663 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.706 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:48.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:48 smithi038 ceph-mon[29779]: pgmap v887: 148 pgs: 148 active+clean; 920 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 182 B/s, 9 objects/s recovering 2024-03-04T16:21:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:48 smithi038 ceph-mon[29779]: osdmap e412: 8 total, 8 up, 7 in 2024-03-04T16:21:48.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:48 smithi038 ceph-mon[38113]: pgmap v887: 148 pgs: 148 active+clean; 920 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 182 B/s, 9 objects/s recovering 2024-03-04T16:21:48.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:48 smithi038 ceph-mon[38113]: osdmap e412: 8 total, 8 up, 7 in 2024-03-04T16:21:48.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:48.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:48.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:48 smithi176 ceph-mon[33619]: pgmap v887: 148 pgs: 148 active+clean; 920 MiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 72 MiB/s wr, 1.15k op/s; 182 B/s, 9 objects/s recovering 2024-03-04T16:21:48.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:48 smithi176 ceph-mon[33619]: osdmap e412: 8 total, 8 up, 7 in 2024-03-04T16:21:48.864 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 12 16 20432 20416 106.319 102.875 0.00587452 0.00939953 2024-03-04T16:21:48.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:48.984 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:48.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:49.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:49.128 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:49.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:49.275 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:49.377 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:49.409 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:49.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:49.544 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:49.643 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:21:49.643 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:21:49.643 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:21:49.643 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:223758 10.d 10:b7113b4c:::benchmark_data_smithi038_155391_object87801:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:06.501685+0000", 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.13782174000001, 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2066799920000002, 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 223758 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.644 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501685+0000", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501685+0000", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501686+0000", 2024-03-04T16:21:49.645 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7400000000000001e-06 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501692+0000", 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.648e-06 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501701+0000", 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1770000000000004e-06 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.646 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501712+0000", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0624e-05 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.502742+0000", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001029812 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.502754+0000", 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2386e-05 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496512+0000", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99375836699999998 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496525+0000", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3047999999999999e-05 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.706986+0000", 2024-03-04T16:21:49.648 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2104606040000001 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.708365+0000", 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001378586 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:223775 10.d 10:b1ab04d8:::benchmark_data_smithi038_155391_object87818:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:06.501725+0000", 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.137781316, 2024-03-04T16:21:49.649 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2213967440000002, 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 223775 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501725+0000", 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.650 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501725+0000", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501726+0000", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1099999999999995e-07 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501729+0000", 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2849999999999999e-06 2024-03-04T16:21:49.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501736+0000", 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6479999999999997e-06 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.501742+0000", 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3500000000000002e-06 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.652 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.502781+0000", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010385430000000001 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.502790+0000", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5689999999999994e-06 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496561+0000", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99377132300000004 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.653 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496570+0000", 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7809999999999994e-06 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.708661+0000", 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.212091257 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.708790+0000", 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000128598 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.654 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.722963+0000", 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014172695000000001 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.723122+0000", 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015898400000000001 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:223784 10.d 10:b33cef20:::benchmark_data_smithi038_155391_object87827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:49.655 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:06.503729+0000", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.13577714500001, 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2769133080000001, 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 223784 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503729+0000", 2024-03-04T16:21:49.656 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503729+0000", 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.657 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.657 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503731+0000", 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0870000000000001e-06 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503737+0000", 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0589999999999999e-06 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503747+0000", 2024-03-04T16:21:49.659 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.5459999999999997e-06 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.503755+0000", 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2500000000000006e-06 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.660 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.504315+0000", 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00055938299999999999 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.504326+0000", 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.172e-05 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496597+0000", 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99227103999999999 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.661 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496606+0000", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7499999999999992e-06 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.708825+0000", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.212218657 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.709800+0000", 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00097528600000000001 2024-03-04T16:21:49.662 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4,7", 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.710135+0000", 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00033474200000000001 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.717535+0000", 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0074004500000000003 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.663 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.718409+0000", 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00087399299999999999 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.721572+0000", 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031632209999999999 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.780445+0000", 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.058872171000000001 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.664 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.780511+0000", 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6793999999999998e-05 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.780642+0000", 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000131159 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.665 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:223797 10.d 10:bfba42a2:::benchmark_data_smithi038_155391_object87840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:06.522368+0000", 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 103.11713870600001, 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2595286620000001, 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 223797 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.666 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522368+0000", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522368+0000", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522370+0000", 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.0889999999999998e-06 2024-03-04T16:21:49.667 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522380+0000", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.013e-05 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522394+0000", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4577e-05 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522404+0000", 2024-03-04T16:21:49.668 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0990000000000005e-06 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522466+0000", 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2192000000000004e-05 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.522478+0000", 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.24e-05 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.669 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496658+0000", 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97418027100000004 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.496667+0000", 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.054e-06 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.714752+0000", 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2180850320000001 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.670 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.714892+0000", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013966099999999999 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,4,7", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.715046+0000", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015363200000000001 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.717811+0000", 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0027655269999999998 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.671 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.718723+0000", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000911928 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.722916+0000", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041930869999999999 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.781719+0000", 2024-03-04T16:21:49.672 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.058802522000000003 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.781779+0000", 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0577e-05 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.781896+0000", 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000116884 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.673 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254086 10.a 10:55f41e7c:::benchmark_data_smithi038_155391_object118129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.465818+0000", 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.173687905999998, 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99007977199999997, 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254086 2024-03-04T16:21:49.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465818+0000", 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465818+0000", 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.675 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465821+0000", 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2120000000000002e-06 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465826+0000", 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.84e-06 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465845+0000", 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8768000000000001e-05 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465854+0000", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2600000000000005e-06 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465907+0000", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3553e-05 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.465916+0000", 2024-03-04T16:21:49.677 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5299999999999996e-06 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.448574+0000", 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98265852600000003 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.448711+0000", 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013722100000000001 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.678 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.448841+0000", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000129715 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.453307+0000", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044656230000000002 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454001+0000", 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00069390499999999998 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.679 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.455816+0000", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0018152400000000001 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.455851+0000", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5142000000000003e-05 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.455898+0000", 2024-03-04T16:21:49.680 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7237000000000002e-05 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254100 10.a 10:53aa8f01:::benchmark_data_smithi038_155391_object118143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.485777+0000", 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.153728999000002, 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97047573499999995, 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.681 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254100 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485777+0000", 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485777+0000", 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.682 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485779+0000", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.773e-06 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485785+0000", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7409999999999998e-06 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485800+0000", 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.5155e-05 2024-03-04T16:21:49.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485809+0000", 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7809999999999994e-06 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485866+0000", 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7241000000000003e-05 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.684 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.485875+0000", 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1859999999999998e-06 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.451234+0000", 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96535897100000001 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.451332+0000", 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7448000000000006e-05 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.685 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.451436+0000", 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010390700000000001 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456105+0000", 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046698030000000001 2024-03-04T16:21:49.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456125+0000", 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9361e-05 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456166+0000", 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1709000000000003e-05 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.687 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.688 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456215+0000", 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8254999999999998e-05 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456253+0000", 2024-03-04T16:21:49.689 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8404e-05 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254102 10.d 10:bdafc075:::benchmark_data_smithi038_155391_object118145:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.486255+0000", 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.153250913999997, 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97005544200000005, 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.690 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254102 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486255+0000", 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.691 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486255+0000", 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486257+0000", 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.497e-06 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.692 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486262+0000", 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1800000000000004e-06 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486278+0000", 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6059e-05 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486287+0000", 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2210000000000003e-06 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.693 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486351+0000", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3563000000000001e-05 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.486359+0000", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1240000000000005e-06 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.449427+0000", 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96306762899999998 2024-03-04T16:21:49.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.449538+0000", 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110985 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.449653+0000", 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000115715 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.695 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454005+0000", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043513010000000001 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454075+0000", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.0388000000000004e-05 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456121+0000", 2024-03-04T16:21:49.696 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0020461759999999998 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456268+0000", 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000146308 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456311+0000", 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3296e-05 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.697 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254105 10.d 10:bc0e89df:::benchmark_data_smithi038_155391_object118148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.488964+0000", 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.150542289999997, 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99425907199999997, 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254105 2024-03-04T16:21:49.698 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488964+0000", 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488964+0000", 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.699 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488966+0000", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.75e-06 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488971+0000", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3820000000000003e-06 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488988+0000", 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6344000000000001e-05 2024-03-04T16:21:49.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.488996+0000", 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2409999999999995e-06 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.489039+0000", 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.3293e-05 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.701 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.489045+0000", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7899999999999996e-06 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.452003+0000", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96295777400000004 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.452106+0000", 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010369699999999999 2024-03-04T16:21:49.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.452347+0000", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024071300000000001 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458463+0000", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0061162200000000003 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458556+0000", 2024-03-04T16:21:49.703 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2726000000000006e-05 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483128+0000", 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024571974 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483178+0000", 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9752000000000001e-05 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.704 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483223+0000", 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5416e-05 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254112 10.d 10:b8d2ebce:::benchmark_data_smithi038_155391_object118155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.492182+0000", 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.147324337999997, 2024-03-04T16:21:49.705 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99112007800000002, 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254112 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492182+0000", 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492182+0000", 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492184+0000", 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7719999999999999e-06 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.707 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492190+0000", 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0299999999999999e-06 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492204+0000", 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4413000000000001e-05 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492227+0000", 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3054e-05 2024-03-04T16:21:49.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492353+0000", 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000126019 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.492362+0000", 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4600000000000003e-06 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.709 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454129+0000", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.961766974 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454235+0000", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010676800000000001 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454369+0000", 2024-03-04T16:21:49.710 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013387800000000001 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458502+0000", 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041326540000000004 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458592+0000", 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0062000000000006e-05 2024-03-04T16:21:49.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483232+0000", 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.024639647000000001 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483269+0000", 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7599999999999999e-05 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.712 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483302+0000", 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2746999999999997e-05 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254123 10.a 10:508fea52:::benchmark_data_smithi038_155391_object118166:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.499902+0000", 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.139604313, 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96063109499999999, 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254123 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499902+0000", 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.714 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499902+0000", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499903+0000", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.029e-06 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499907+0000", 2024-03-04T16:21:49.715 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8489999999999999e-06 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499914+0000", 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.559e-06 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.499921+0000", 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3049999999999998e-06 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.716 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.501625+0000", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017043410000000001 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.501638+0000", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.322e-05 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454040+0000", 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95240128800000001 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.717 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454128+0000", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7998999999999996e-05 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.454204+0000", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6330999999999998e-05 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458369+0000", 2024-03-04T16:21:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041647409999999996 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.460365+0000", 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001996027 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.460416+0000", 2024-03-04T16:21:49.719 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0856999999999998e-05 2024-03-04T16:21:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:49.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.460470+0000", 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3992000000000001e-05 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.721 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.460533+0000", 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.3557000000000002e-05 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254142 10.d 10:ba5324d7:::benchmark_data_smithi038_155391_object118185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.516385+0000", 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.123121769000001, 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96699146499999999, 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.722 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254142 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516385+0000", 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.723 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516385+0000", 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516386+0000", 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.903e-06 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.724 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516393+0000", 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2940000000000002e-06 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516409+0000", 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6172e-05 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516418+0000", 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2320000000000007e-06 2024-03-04T16:21:49.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516477+0000", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8934999999999997e-05 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.516486+0000", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8910000000000001e-06 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456405+0000", 2024-03-04T16:21:49.726 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93991908899999999 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456507+0000", 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010186199999999999 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456620+0000", 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011283800000000001 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.727 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.464779+0000", 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0081590099999999995 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.464814+0000", 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5366e-05 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483308+0000", 2024-03-04T16:21:49.728 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018493880000000001 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483344+0000", 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.625e-05 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483376+0000", 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1742999999999997e-05 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254145 10.a 10:5d6910a0:::benchmark_data_smithi038_155391_object118188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.519005+0000", 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.120501029000003, 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94479666699999998, 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254145 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.730 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519005+0000", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519005+0000", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519007+0000", 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9180000000000001e-06 2024-03-04T16:21:49.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519014+0000", 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6880000000000002e-06 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519030+0000", 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6132000000000001e-05 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.732 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519039+0000", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3270000000000007e-06 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519095+0000", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5441e-05 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519104+0000", 2024-03-04T16:21:49.733 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.7700000000000007e-06 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456320+0000", 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.937216402 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456524+0000", 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000204172 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 2,5", 2024-03-04T16:21:49.734 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.456643+0000", 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011855599999999999 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.460378+0000", 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037349760000000001 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.463663+0000", 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032851489999999998 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.735 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.463698+0000", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4934999999999997e-05 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.463755+0000", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7361000000000002e-05 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.463802+0000", 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6839999999999999e-05 2024-03-04T16:21:49.736 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254148 10.d 10:bc861580:::benchmark_data_smithi038_155391_object118191:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.519926+0000", 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.119580470000002, 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96352369100000002, 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.737 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254148 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519926+0000", 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519926+0000", 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519927+0000", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.392e-06 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519933+0000", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6969999999999998e-06 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519956+0000", 2024-03-04T16:21:49.739 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2636000000000001e-05 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.519965+0000", 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0100000000000001e-06 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.520038+0000", 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3236000000000002e-05 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.740 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.520047+0000", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.1470000000000007e-06 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.458620+0000", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93857338400000001 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.462278+0000", 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0036574810000000002 2024-03-04T16:21:49.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.462367+0000", 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9139000000000002e-05 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472520+0000", 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010152858000000001 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.742 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472562+0000", 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2206999999999998e-05 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483382+0000", 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010819535 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483418+0000", 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6436999999999997e-05 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.743 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483450+0000", 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.1532000000000002e-05 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254155 10.d 10:b02113b2:::benchmark_data_smithi038_155391_object118198:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.527638+0000", 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.111868125999997, 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95588486699999997, 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.744 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254155 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527638+0000", 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.745 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527638+0000", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527640+0000", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7e-06 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527645+0000", 2024-03-04T16:21:49.746 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0390000000000004e-06 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527657+0000", 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1659e-05 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527663+0000", 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5030000000000002e-06 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.747 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527747+0000", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3442999999999994e-05 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.527756+0000", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.2799999999999992e-06 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.464840+0000", 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93708367400000003 2024-03-04T16:21:49.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.464954+0000", 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114764 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.465067+0000", 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112577 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.749 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472600+0000", 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0075328779999999998 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.477436+0000", 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0048360879999999997 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483455+0000", 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060194999999999997 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483490+0000", 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.4961e-05 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483523+0000", 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.2801e-05 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:254164 10.d 10:b60ee48c:::benchmark_data_smithi038_155391_object118207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:48.536976+0000", 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 61.102530629999997, 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94741540400000002, 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.752 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 254164 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.536976+0000", 2024-03-04T16:21:49.753 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.536976+0000", 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.536978+0000", 2024-03-04T16:21:49.754 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.351e-06 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.536985+0000", 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7560000000000002e-06 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.536999+0000", 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4017e-05 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.755 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.537007+0000", 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2050000000000002e-06 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.537053+0000", 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5936999999999997e-05 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.756 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.537060+0000", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.613e-06 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472632+0000", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93557238700000001 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472732+0000", 2024-03-04T16:21:49.757 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000100008 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,7", 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.472841+0000", 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000109425 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.480499+0000", 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076572760000000002 2024-03-04T16:21:49.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.483529+0000", 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030301450000000001 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.484299+0000", 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00076974300000000003 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.759 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.484351+0000", 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2262000000000002e-05 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:49.484391+0000", 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0278999999999998e-05 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.760 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:16721 11.8 11:18fe777e:::benchmark_data_smithi038_182448_object16720:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e409)", 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:21:45.397933+0000", 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2415735870000004, 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1301748279999999, 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:21:49.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16721 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397933+0000", 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397933+0000", 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.762 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397935+0000", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.3649999999999998e-06 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397972+0000", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6875000000000002e-05 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397985+0000", 2024-03-04T16:21:49.763 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.296e-05 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.397992+0000", 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3699999999999997e-06 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398053+0000", 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0739999999999998e-05 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.764 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398060+0000", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1720000000000001e-06 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402662+0000", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0046013010000000003 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402671+0000", 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6509999999999996e-06 2024-03-04T16:21:49.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.516177+0000", 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1135059650000001 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.516629+0000", 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00045207 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.766 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.516776+0000", 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014680800000000001 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.521943+0000", 2024-03-04T16:21:49.767 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051671640000000001 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.522079+0000", 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013557700000000001 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.522130+0000", 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1549999999999999e-05 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.768 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.526847+0000", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.004716568 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.526902+0000", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5479999999999997e-05 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.528108+0000", 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001205212 2024-03-04T16:21:49.769 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:16735 11.1 11:8c3bb356:::benchmark_data_smithi038_182448_object16734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e409)", 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:21:45.398005+0000", 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.24150092, 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1233048649999999, 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:21:49.770 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16735 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398005+0000", 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398005+0000", 2024-03-04T16:21:49.771 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398006+0000", 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4399999999999999e-07 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398023+0000", 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6748999999999998e-05 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.772 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398028+0000", 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.848e-06 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398032+0000", 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.968e-06 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.773 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398079+0000", 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6856e-05 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op must wait for map", 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398085+0000", 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0689999999999998e-06 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.403818+0000", 2024-03-04T16:21:49.774 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0057333510000000002 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.403827+0000", 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0180000000000007e-06 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.515926+0000", 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.112098676 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.775 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.516451+0000", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00052558800000000001 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.516623+0000", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000171316 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519056+0000", 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024337349999999998 2024-03-04T16:21:49.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.520338+0000", 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0012818339999999999 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.520397+0000", 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8434999999999999e-05 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.777 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.520503+0000", 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010685 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.521310+0000", 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00080682799999999999 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.778 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:16736 11.8 11:1e776209:::benchmark_data_smithi038_182448_object16735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e409)", 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:21:45.398042+0000", 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2414642779999996, 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.130969763, 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16736 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398042+0000", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398042+0000", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398043+0000", 2024-03-04T16:21:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4899999999999995e-07 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398064+0000", 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.1580999999999999e-05 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398073+0000", 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.602e-06 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398080+0000", 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6889999999999998e-06 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398108+0000", 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.7928000000000001e-05 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398111+0000", 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.3100000000000001e-06 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402695+0000", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045845820000000002 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402703+0000", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.61e-06 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519075+0000", 2024-03-04T16:21:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116372285 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519234+0000", 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015883600000000001 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519404+0000", 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00017019800000000001 2024-03-04T16:21:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.522034+0000", 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0026293219999999999 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.526481+0000", 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044473660000000003 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.526798+0000", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00031710899999999999 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.528313+0000", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001514926 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.528413+0000", 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9618999999999996e-05 2024-03-04T16:21:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.529012+0000", 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00059915100000000002 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:16737 11.1 11:83673a54:::benchmark_data_smithi038_182448_object16736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e409)", 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:21:45.398091+0000", 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.241414926, 2024-03-04T16:21:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1290591349999999, 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16737 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398091+0000", 2024-03-04T16:21:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398091+0000", 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398092+0000", 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8800000000000002e-07 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398115+0000", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2972e-05 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398124+0000", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4440000000000008e-06 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398133+0000", 2024-03-04T16:21:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7499999999999998e-06 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398183+0000", 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9524000000000002e-05 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398189+0000", 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7150000000000003e-06 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.403852+0000", 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0056630409999999997 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.403858+0000", 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7660000000000001e-06 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519106+0000", 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1152473389999999 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519253+0000", 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014767400000000001 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,6", 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.519407+0000", 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015389099999999999 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.521749+0000", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0023412540000000001 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.522410+0000", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000661632 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.525605+0000", 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0031946750000000001 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.525712+0000", 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010723 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.527151+0000", 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00143844 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:16738 11.8 11:1c4ba712:::benchmark_data_smithi038_182448_object16737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e409)", 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:21:45.398142+0000", 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 4.2413639720000003, 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.131642842, 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 16738 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:21:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398142+0000", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398142+0000", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398143+0000", 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9100000000000003e-07 2024-03-04T16:21:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398169+0000", 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6191999999999999e-05 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398178+0000", 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5229999999999995e-06 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398185+0000", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6699999999999997e-06 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398241+0000", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6096e-05 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting_for_map not empty", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.398246+0000", 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.8270000000000003e-06 2024-03-04T16:21:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402728+0000", 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0044814360000000001 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:45.402735+0000", 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7939999999999997e-06 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.520354+0000", 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1176195849999999 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.520506+0000", 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015164700000000001 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 3,5,6", 2024-03-04T16:21:49.801 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.521075+0000", 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056915100000000005 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.522144+0000", 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0010694120000000001 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.528165+0000", 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0060207250000000002 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.528263+0000", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7961999999999995e-05 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.529170+0000", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00090684700000000001 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.529298+0000", 2024-03-04T16:21:49.803 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000128176 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:21:46.529785+0000", 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00048700799999999998 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:21:49.804 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:21:49.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:49.859 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:49.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:49.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 13 16 22006 21990 105.685 98.375 0.00689699 0.0094527 2024-03-04T16:21:49.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:49.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:49 smithi038 ceph-mon[29779]: Health check failed: Degraded data redundancy: 3163/61131 objects degraded (5.174%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:49 smithi038 ceph-mon[38113]: Health check failed: Degraded data redundancy: 3163/61131 objects degraded (5.174%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:50.004 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:50.105 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:50.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:49 smithi176 ceph-mon[33619]: Health check failed: Degraded data redundancy: 3163/61131 objects degraded (5.174%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:50.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:50.239 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:50.273 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:50.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:50.375 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:50.416 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:50.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:50.557 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:21:50.557 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 1 2024-03-04T16:21:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.559 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:50.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:50.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.707 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:50.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:50.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:50.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:50.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 14 16 23524 23508 104.912 94.875 0.0176372 0.00952365 2024-03-04T16:21:50.945 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:50.971 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:50.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:51.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.136 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:51.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:51.158 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:21:51.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:51.267 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:51.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:21:51.369 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:51.398 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:51.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.527 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:51.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:51.653 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:51.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:51 smithi038 ceph-mon[29779]: pgmap v889: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.27k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 174 objects/s recovering 2024-03-04T16:21:51.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:51 smithi038 ceph-mon[29779]: osdmap e413: 8 total, 8 up, 7 in 2024-03-04T16:21:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:51 smithi038 ceph-mon[38113]: pgmap v889: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.27k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 174 objects/s recovering 2024-03-04T16:21:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:51 smithi038 ceph-mon[38113]: osdmap e413: 8 total, 8 up, 7 in 2024-03-04T16:21:51.755 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:51.780 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:51.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:51 smithi176 ceph-mon[33619]: pgmap v889: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 950 B/s rd, 79 MiB/s wr, 1.27k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 174 objects/s recovering 2024-03-04T16:21:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:51 smithi176 ceph-mon[33619]: osdmap e413: 8 total, 8 up, 7 in 2024-03-04T16:21:51.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 15 16 24392 24376 101.535 54.25 0.00566274 0.00961936 2024-03-04T16:21:51.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:51.907 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:51.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:52.009 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:52.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:52.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:52.163 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:52.265 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:52.289 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:52.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:52.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:52 smithi038 ceph-mon[29779]: osdmap e414: 8 total, 8 up, 7 in 2024-03-04T16:21:52.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:52 smithi038 ceph-mon[29779]: pgmap v892: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 954 B/s rd, 80 MiB/s wr, 1.28k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 175 objects/s recovering 2024-03-04T16:21:52.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:52 smithi038 ceph-mon[38113]: osdmap e414: 8 total, 8 up, 7 in 2024-03-04T16:21:52.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:52 smithi038 ceph-mon[38113]: pgmap v892: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 954 B/s rd, 80 MiB/s wr, 1.28k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 175 objects/s recovering 2024-03-04T16:21:52.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:52.555 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:52.656 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:52.686 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:52.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:52.831 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:52.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:52 smithi176 ceph-mon[33619]: osdmap e414: 8 total, 8 up, 7 in 2024-03-04T16:21:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:52 smithi176 ceph-mon[33619]: pgmap v892: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 peering, 139 active+clean; 1.2 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 954 B/s rd, 80 MiB/s wr, 1.28k op/s; 3163/61131 objects degraded (5.174%); 4487/61131 objects misplaced (7.340%); 9.7 MiB/s, 175 objects/s recovering 2024-03-04T16:21:52.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 16 16 25045 25029 97.7403 40.8125 0.00791377 0.0102244 2024-03-04T16:21:52.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:52.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:53.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:53.133 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:53.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:53.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:53.363 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.387 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:53.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:53.488 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:53.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[29779]: osdmap e415: 8 total, 8 up, 7 in 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[38113]: osdmap e415: 8 total, 8 up, 7 in 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:53.493 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:21:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:21:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:21:53.529 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:53.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:53.630 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:53.655 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:53.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:53.783 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:53.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:53.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:21:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:21:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:53 smithi176 ceph-mon[33619]: osdmap e415: 8 total, 8 up, 7 in 2024-03-04T16:21:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:53 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:21:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:21:53.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 17 16 25921 25905 95.2116 54.75 0.00259599 0.0102919 2024-03-04T16:21:53.885 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:53.908 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:53.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:54.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:54.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:54.138 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:54.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:54.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:54.295 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:54.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:54.420 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:54.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:54.550 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:54.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:54.614 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[29779]: osdmap e416: 8 total, 8 up, 7 in 2024-03-04T16:21:54.614 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[29779]: pgmap v895: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 141 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 597 op/s; 3163/68289 objects degraded (4.632%); 2861/68289 objects misplaced (4.190%); 25 MiB/s, 406 objects/s recovering 2024-03-04T16:21:54.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:54.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[38113]: osdmap e416: 8 total, 8 up, 7 in 2024-03-04T16:21:54.614 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:54 smithi038 ceph-mon[38113]: pgmap v895: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 141 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 597 op/s; 3163/68289 objects degraded (4.632%); 2861/68289 objects misplaced (4.190%); 25 MiB/s, 406 objects/s recovering 2024-03-04T16:21:54.651 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:54.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:54.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:21:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:54 smithi176 ceph-mon[33619]: osdmap e416: 8 total, 8 up, 7 in 2024-03-04T16:21:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:54 smithi176 ceph-mon[33619]: pgmap v895: 148 pgs: 2 active+recovery_wait+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 141 active+clean; 1.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 37 MiB/s wr, 597 op/s; 3163/68289 objects degraded (4.632%); 2861/68289 objects misplaced (4.190%); 25 MiB/s, 406 objects/s recovering 2024-03-04T16:21:54.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 18 16 26179 26163 90.8187 16.125 0.002482 0.0108532 2024-03-04T16:21:54.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:54.928 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:54.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:55.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:55.058 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:55.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:55.184 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:55.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:55.286 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:55.311 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:21:55.393 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:21:55.394 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:21:55.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:55.450 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:55.552 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:55.588 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:55.690 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:55.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:55.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: osdmap e417: 8 total, 8 up, 7 in 2024-03-04T16:21:55.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/684860780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:55.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: Health check update: Degraded data redundancy: 3163/68289 objects degraded (4.632%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[29779]: osdmap e418: 8 total, 8 up, 7 in 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: osdmap e417: 8 total, 8 up, 7 in 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/684860780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: Health check update: Degraded data redundancy: 3163/68289 objects degraded (4.632%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:55.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:55 smithi038 ceph-mon[38113]: osdmap e418: 8 total, 8 up, 7 in 2024-03-04T16:21:55.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:55.842 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:55.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:55.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: osdmap e417: 8 total, 8 up, 7 in 2024-03-04T16:21:55.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/684860780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:21:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:21:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:21:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: Health check update: Degraded data redundancy: 3163/68289 objects degraded (4.632%), 3 pgs degraded (PG_DEGRADED) 2024-03-04T16:21:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:55 smithi176 ceph-mon[33619]: osdmap e418: 8 total, 8 up, 7 in 2024-03-04T16:21:55.867 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 19 16 26505 26489 87.1117 20.375 0.00358105 0.0112856 2024-03-04T16:21:55.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:55.970 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:55.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:56.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:56.098 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:56.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:56.226 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:56.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:56.327 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:56.353 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:56.455 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:56.498 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:56.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:56.600 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:56.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:56.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:56.765 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:56.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:56.868 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:21:56.868871+0000 min lat: 0.00129433 max lat: 1.98384 avg lat: 0.0117731 2024-03-04T16:21:56.868 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:21:56.868 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 20 16 27187 27171 84.8875 42.625 0.00916776 0.0117731 2024-03-04T16:21:56.908 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:56.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:57.010 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:57.053 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:57.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:57.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:57.199 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:57.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:57.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:57.339 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:57.441 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:57.472 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:57.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:57.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[29779]: pgmap v898: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1025/77313 objects degraded (1.326%); 3177/77313 objects misplaced (4.109%); 54 MiB/s, 861 objects/s recovering 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[29779]: osdmap e419: 8 total, 8 up, 7 in 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[38113]: pgmap v898: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1025/77313 objects degraded (1.326%); 3177/77313 objects misplaced (4.109%); 54 MiB/s, 861 objects/s recovering 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:21:57.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:57 smithi038 ceph-mon[38113]: osdmap e419: 8 total, 8 up, 7 in 2024-03-04T16:21:57.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:57.604 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:57.706 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:57.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:57.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:57.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:57 smithi176 ceph-mon[33619]: pgmap v898: 148 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 141 active+clean; 1.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.35k op/s; 1025/77313 objects degraded (1.326%); 3177/77313 objects misplaced (4.109%); 54 MiB/s, 861 objects/s recovering 2024-03-04T16:21:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:57 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:21:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:57 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:21:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:57 smithi176 ceph-mon[33619]: osdmap e419: 8 total, 8 up, 7 in 2024-03-04T16:21:57.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:57.868 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 21 16 28422 28406 84.5206 77.1875 0.00229959 0.0116602 2024-03-04T16:21:57.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:57.990 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:57.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:58.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:58.118 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:58.223 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:58.248 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:58.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:58.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:58.375 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:58.476 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:58.507 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:58.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:21:58.633 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:21:58.735 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:58.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:21:58.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:21:58.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[29779]: osdmap e420: 8 total, 8 up, 7 in 2024-03-04T16:21:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:21:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:21:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:58 smithi038 ceph-mon[38113]: osdmap e420: 8 total, 8 up, 7 in 2024-03-04T16:21:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.759 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:58.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:58.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:58 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:21:58.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:58 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:21:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:58 smithi176 ceph-mon[33619]: osdmap e420: 8 total, 8 up, 7 in 2024-03-04T16:21:58.860 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:21:58.868 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 22 16 28541 28525 81.0174 7.4375 0.00215385 0.0121407 2024-03-04T16:21:58.883 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:58.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:21:58.985 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:21:59.009 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:59.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:21:59.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:21:59.134 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:21:59.235 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:59.259 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:59.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:59.361 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:21:59.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:21:59.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:21:59.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[29779]: pgmap v901: 148 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1025/78090 objects degraded (1.313%); 2014/78090 objects misplaced (2.579%); 47 MiB/s, 746 objects/s recovering 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[29779]: osdmap e421: 8 total, 8 up, 7 in 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[38113]: pgmap v901: 148 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1025/78090 objects degraded (1.313%); 2014/78090 objects misplaced (2.579%); 47 MiB/s, 746 objects/s recovering 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[38113]: osdmap e421: 8 total, 8 up, 7 in 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:21:59.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:21:59 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:21:59.518 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:21:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:21:59.534 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:59.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:59.664 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:21:59.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:21:59.784 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:21:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:59.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:21:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:59 smithi176 ceph-mon[33619]: pgmap v901: 148 pgs: 1 creating+peering, 1 active+recovery_wait+remapped, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 142 active+clean; 1.6 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 1025/78090 objects degraded (1.313%); 2014/78090 objects misplaced (2.579%); 47 MiB/s, 746 objects/s recovering 2024-03-04T16:21:59.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:59 smithi176 ceph-mon[33619]: osdmap e421: 8 total, 8 up, 7 in 2024-03-04T16:21:59.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:59 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:21:59.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:21:59 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:21:59.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 23 16 29161 29145 79.1671 38.75 0.00780462 0.0126238 2024-03-04T16:21:59.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:21:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:21:59.968 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:21:59.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:00.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:00.099 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:00.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:00.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:00.384 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:00.415 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T16:22:00.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:00.527 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:00.629 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:00.658 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: osdmap e422: 8 total, 8 up, 7 in 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: pgmap v904: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 674 op/s; 1783/85392 objects misplaced (2.088%); 38 MiB/s, 604 objects/s recovering 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2789930375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: Health check update: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:22:00.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[29779]: osdmap e423: 8 total, 8 up, 7 in 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: osdmap e422: 8 total, 8 up, 7 in 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: pgmap v904: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 674 op/s; 1783/85392 objects misplaced (2.088%); 38 MiB/s, 604 objects/s recovering 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2789930375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: Health check update: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:22:00.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:00 smithi038 ceph-mon[38113]: osdmap e423: 8 total, 8 up, 7 in 2024-03-04T16:22:00.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:00.801 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: osdmap e422: 8 total, 8 up, 7 in 2024-03-04T16:22:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: pgmap v904: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 42 MiB/s wr, 674 op/s; 1783/85392 objects misplaced (2.088%); 38 MiB/s, 604 objects/s recovering 2024-03-04T16:22:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:22:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:22:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2789930375' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: from='client.35879 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:22:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: Health check update: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded (PG_DEGRADED) 2024-03-04T16:22:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: from='client.35879 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:22:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:00 smithi176 ceph-mon[33619]: osdmap e423: 8 total, 8 up, 7 in 2024-03-04T16:22:00.871 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 24 16 30810 30794 80.1621 103.062 0.0138158 0.0124688 2024-03-04T16:22:00.903 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:00.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:00.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:01.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:01.074 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:01.177 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:01.208 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:01.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:22:01.310 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.358 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:01.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:01.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 1 1 2024-03-04T16:22:01.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded) 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:22:01.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:22:01.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded) 2024-03-04T16:22:01.558 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:22:01.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:22:01.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:22:01.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:22:01.559 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:01 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:22:01.590 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:01.642 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:01.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:01.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.806 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:01.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1025/78090 objects degraded (1.313%), 1 pg degraded) 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:22:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:01 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:22:01.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 25 16 32454 32438 81.0648 102.75 0.00610399 0.0123283 2024-03-04T16:22:01.907 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:01.950 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:01.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:02.052 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:02.101 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:02.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:02.243 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:02.345 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:02.375 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:02.477 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:02.517 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:02.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:02.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:02.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:02 smithi038 ceph-mon[29779]: pgmap v906: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 633 op/s; 1783/85392 objects misplaced (2.088%); 36 MiB/s, 568 objects/s recovering 2024-03-04T16:22:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:02 smithi038 ceph-mon[38113]: pgmap v906: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 633 op/s; 1783/85392 objects misplaced (2.088%); 36 MiB/s, 568 objects/s recovering 2024-03-04T16:22:02.762 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:02.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:02.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:02 smithi176 ceph-mon[33619]: pgmap v906: 148 pgs: 1 active+recovering+remapped, 1 creating+peering, 2 active+recovering+undersized+remapped, 144 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 40 MiB/s wr, 633 op/s; 1783/85392 objects misplaced (2.088%); 36 MiB/s, 568 objects/s recovering 2024-03-04T16:22:02.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 26 16 33995 33979 81.6507 96.3125 0.00586327 0.0122406 2024-03-04T16:22:02.887 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:02.933 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:02.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:03.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:03.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:03.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:03.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:03.204 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:03.306 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:03.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:03.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:03.475 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:03.576 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:03.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:03.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:22:03.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:22:03.697 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:03.707 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:03.730 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:03.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:03.758 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:03 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:22:03.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:03 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:22:03.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:03 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:22:03.759 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:03 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:22:03.759 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:03.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:03.856 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:03.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 27 16 35158 35142 81.3186 72.6875 0.00201955 0.0121614 2024-03-04T16:22:03.958 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:03.984 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:03.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:04.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:04.109 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:04.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:04.211 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:04.236 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:04.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:04.361 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:04.463 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:04.487 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:04.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:04.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:04.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:04.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:04.715 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:04.739 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:04.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:04.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:04 smithi176 ceph-mon[33619]: osdmap e424: 8 total, 8 up, 7 in 2024-03-04T16:22:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:04 smithi176 ceph-mon[33619]: pgmap v908: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 975 B/s rd, 53 MiB/s wr, 848 op/s; 568/95220 objects misplaced (0.597%); 20 MiB/s, 323 objects/s recovering 2024-03-04T16:22:04.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 28 16 35298 35282 78.7275 8.75 0.0121432 0.0126963 2024-03-04T16:22:04.880 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:04.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:04.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[29779]: osdmap e424: 8 total, 8 up, 7 in 2024-03-04T16:22:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[29779]: pgmap v908: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 975 B/s rd, 53 MiB/s wr, 848 op/s; 568/95220 objects misplaced (0.597%); 20 MiB/s, 323 objects/s recovering 2024-03-04T16:22:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[38113]: osdmap e424: 8 total, 8 up, 7 in 2024-03-04T16:22:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:04 smithi038 ceph-mon[38113]: pgmap v908: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1.9 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 975 B/s rd, 53 MiB/s wr, 848 op/s; 568/95220 objects misplaced (0.597%); 20 MiB/s, 323 objects/s recovering 2024-03-04T16:22:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:05.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:05.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:05.147 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:05.249 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:05.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:05.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:05.418 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:05.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:05.549 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:05.651 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:05.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:05.772 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:22:05.772 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: osdmap e425: 8 total, 8 up, 7 in 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:22:05.773 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:05 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:22:05.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:05.812 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:05.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 29 16 36778 36762 79.2019 92.5 0.00211915 0.012613 2024-03-04T16:22:05.914 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:05.937 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:05.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:05.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: osdmap e425: 8 total, 8 up, 7 in 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:22:05.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: osdmap e425: 8 total, 8 up, 7 in 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:22:05.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:05 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:22:06.039 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:06.072 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:06.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:06.200 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:06.302 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:06.342 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:06.444 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:06.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:06.580 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:06.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:06.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:06.757 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:06.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:06 smithi176 ceph-mon[33619]: pgmap v910: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s; 568/102621 objects misplaced (0.553%); 13 MiB/s, 202 objects/s recovering 2024-03-04T16:22:06.859 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:06.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 30 16 38232 38216 79.5907 90.875 0.0226211 0.012557 2024-03-04T16:22:06.888 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:06.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:06.990 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:06.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:06 smithi038 ceph-mon[29779]: pgmap v910: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s; 568/102621 objects misplaced (0.553%); 13 MiB/s, 202 objects/s recovering 2024-03-04T16:22:06.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:06 smithi038 ceph-mon[38113]: pgmap v910: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 60 MiB/s wr, 957 op/s; 568/102621 objects misplaced (0.553%); 13 MiB/s, 202 objects/s recovering 2024-03-04T16:22:07.013 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:07.115 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.157 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:07.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:07.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:07.302 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:07.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:07.432 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:07.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:07.570 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:07.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:07.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:07.722 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:07.824 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 5.1b deep-scrub starts 2024-03-04T16:22:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:07 smithi176 ceph-mon[33619]: 5.1b deep-scrub ok 2024-03-04T16:22:07.872 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 31 16 39768 39752 80.1192 96 0.00648776 0.0124747 2024-03-04T16:22:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:07.876 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:07.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:07.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 5.1b deep-scrub starts 2024-03-04T16:22:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[29779]: 5.1b deep-scrub ok 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 5.1b deep-scrub starts 2024-03-04T16:22:07.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:07 smithi038 ceph-mon[38113]: 5.1b deep-scrub ok 2024-03-04T16:22:08.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:08.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:08.165 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:08.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:08.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:08.406 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:08.437 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:08.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:08.572 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:08.674 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:08.705 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:08.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[29779]: osdmap e426: 8 total, 8 up, 7 in 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[29779]: pgmap v912: 148 pgs: 148 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.24k op/s; 19 MiB/s, 297 objects/s recovering 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[38113]: osdmap e426: 8 total, 8 up, 7 in 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[38113]: pgmap v912: 148 pgs: 148 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.24k op/s; 19 MiB/s, 297 objects/s recovering 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:08.831 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:08.838 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:08 smithi176 ceph-mon[33619]: osdmap e426: 8 total, 8 up, 7 in 2024-03-04T16:22:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:08 smithi176 ceph-mon[33619]: pgmap v912: 148 pgs: 148 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 78 MiB/s wr, 1.24k op/s; 19 MiB/s, 297 objects/s recovering 2024-03-04T16:22:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:08.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:08.873 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 32 16 41377 41361 80.7577 100.562 0.0108787 0.0123777 2024-03-04T16:22:08.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:08.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:08.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:09.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:09.114 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:09.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:09.247 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:09.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:09.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:09.480 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:09.512 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:09.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:09.614 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:09.643 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:09.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:09.784 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:09.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:09 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:22:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:09 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:22:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:09 smithi176 ceph-mon[33619]: 5.2 deep-scrub starts 2024-03-04T16:22:09.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:09 smithi176 ceph-mon[33619]: 5.2 deep-scrub ok 2024-03-04T16:22:09.873 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 33 16 42978 42962 81.3423 100.062 0.0050409 0.0122882 2024-03-04T16:22:09.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:09.927 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:09.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[29779]: 5.2 deep-scrub starts 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[29779]: 5.2 deep-scrub ok 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[38113]: 5.2 deep-scrub starts 2024-03-04T16:22:09.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:09 smithi038 ceph-mon[38113]: 5.2 deep-scrub ok 2024-03-04T16:22:10.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:10.070 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:10.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:10.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:10.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:10.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:10.436 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:10.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.478 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:10.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:10.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:10.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:10.723 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:10.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:10.849 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: pgmap v913: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 822 B/s rd, 89 MiB/s wr, 1.43k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:22:10.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:10 smithi176 ceph-mon[33619]: osdmap e427: 8 total, 8 up, 7 in 2024-03-04T16:22:10.873 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 34 16 44242 44226 81.2731 79 0.00851687 0.0122986 2024-03-04T16:22:10.887 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:10.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:10.951 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: pgmap v913: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 822 B/s rd, 89 MiB/s wr, 1.43k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:22:10.952 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[29779]: osdmap e427: 8 total, 8 up, 7 in 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: pgmap v913: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 822 B/s rd, 89 MiB/s wr, 1.43k op/s; 5.7 MiB/s, 91 objects/s recovering 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:22:10.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:10 smithi038 ceph-mon[38113]: osdmap e427: 8 total, 8 up, 7 in 2024-03-04T16:22:10.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:11.030 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:11.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:11.162 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:11.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:11.307 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:11.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:11.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:11.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:11.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:11.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:11.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:11.620 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:22:11.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:11.722 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:11.824 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:11.855 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:11.873 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 35 16 45738 45722 81.6222 93.5 0.00827977 0.0122455 2024-03-04T16:22:11.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:22:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:22:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2989907832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: from='client.35885 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2989907832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: from='client.35885 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:11 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:22:11.996 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:11.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:12.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:12.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:22:12.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:22:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:22:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2989907832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: from='client.35885 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:22:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:11 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:22:12.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:12.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:12.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:12.275 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:12.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.417 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:12.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:12.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:12.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:12.647 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:12.674 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:12.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:12.873 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 36 16 47205 47189 81.9015 91.6875 0.00833356 0.0122041 2024-03-04T16:22:12.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[29779]: from='client.35885 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[29779]: osdmap e428: 8 total, 8 up, 7 in 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[29779]: pgmap v916: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[38113]: from='client.35885 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:22:12.910 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[38113]: osdmap e428: 8 total, 8 up, 7 in 2024-03-04T16:22:12.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:12 smithi038 ceph-mon[38113]: pgmap v916: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-04T16:22:12.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:12.962 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:12.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:13.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:13.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:13.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:13.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:12 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:13.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:12 smithi176 ceph-mon[33619]: from='client.35885 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:22:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:12 smithi176 ceph-mon[33619]: osdmap e428: 8 total, 8 up, 7 in 2024-03-04T16:22:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:12 smithi176 ceph-mon[33619]: pgmap v916: 148 pgs: 148 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 67 MiB/s wr, 1.07k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-04T16:22:13.134 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:22:13.134 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:22:13.207 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:13.254 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:13.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:13.381 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:13.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:13.494 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:13.521 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:13.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:13.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:13.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:13.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.796 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:13.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:13.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:13.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 37 16 48631 48615 82.0954 89.125 0.0145478 0.0121751 2024-03-04T16:22:13.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:13.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:13.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:13.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:14.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:14.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:14.187 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.225 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:14.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:14.326 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:14.358 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:14.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:14.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:14.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:14.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:14.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:14.624 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:14.723 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:22:14.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:14.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:22:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.812 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:14.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:14.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 38 16 50071 50055 82.3033 90 0.0174523 0.0121438 2024-03-04T16:22:14.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:14.950 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:14.950 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:14 smithi038 ceph-mon[38113]: pgmap v917: 148 pgs: 148 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 5.8 MiB/s, 92 objects/s recovering 2024-03-04T16:22:14.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:14.950 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:14 smithi038 ceph-mon[29779]: pgmap v917: 148 pgs: 148 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 5.8 MiB/s, 92 objects/s recovering 2024-03-04T16:22:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:14.964 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:14.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:15.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:15.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:15.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:14 smithi176 ceph-mon[33619]: pgmap v917: 148 pgs: 148 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 5.8 MiB/s, 92 objects/s recovering 2024-03-04T16:22:15.115 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:15.216 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:15.243 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:15.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:15.389 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:15.491 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.538 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:15.639 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:15.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:15.782 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:15.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:15.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 39 15 51424 51409 82.3626 84.625 0.0168549 0.0121358 2024-03-04T16:22:15.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:15.965 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:15.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:16.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:16.100 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:16.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:16.237 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:16.339 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:16.369 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:16.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:16.492 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:16.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:16.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[38113]: osdmap e429: 8 total, 8 up, 7 in 2024-03-04T16:22:16.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:16.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:16.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:16 smithi038 ceph-mon[29779]: osdmap e429: 8 total, 8 up, 7 in 2024-03-04T16:22:16.518 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:16.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:16.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.670 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:16.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:16.771 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:16.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:16 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:16 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:16 smithi176 ceph-mon[33619]: osdmap e429: 8 total, 8 up, 7 in 2024-03-04T16:22:16.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:22:16.874959+0000 min lat: 0.00129433 max lat: 1.98384 avg lat: 0.0121303 2024-03-04T16:22:16.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:22:16.874 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 40 16 52765 52749 82.3972 83.75 0.0249314 0.0121303 2024-03-04T16:22:16.902 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:16.933 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:16.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:17.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:17.061 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:17.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:17.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:17.205 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:17.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:17.307 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:17.335 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:17.437 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:17.467 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:17.490 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:17 smithi038 ceph-mon[29779]: pgmap v919: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:22:17.490 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2208546502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:17.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:17 smithi038 ceph-mon[38113]: pgmap v919: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:22:17.490 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2208546502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:17.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:17.608 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:17.710 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:17.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:22:17.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:22:17.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:22:17.738 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:22:17.739 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:22:17.739 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:22:17.739 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:22:17.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:22:17.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:17.759 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:22:17.759 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:22:17.759 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:22:17.759 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:22:17.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:22:17.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:22:17.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:22:17.760 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:22:17.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:17.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:17 smithi176 ceph-mon[33619]: pgmap v919: 148 pgs: 148 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-04T16:22:17.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2208546502' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:17.875 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 41 16 54068 54052 82.372 81.4375 0.0122858 0.0121344 2024-03-04T16:22:17.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:17.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:17.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:18.030 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:18.132 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:18.172 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:18.276 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:18.315 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:18.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:18.458 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:18.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:18.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:18.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:18.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:18 smithi038 ceph-mon[29779]: pgmap v920: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 700 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:22:18.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:18 smithi038 ceph-mon[38113]: pgmap v920: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 700 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:22:18.783 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:18.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:18.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:18 smithi176 ceph-mon[33619]: pgmap v920: 148 pgs: 148 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 700 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-04T16:22:18.875 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 42 16 55409 55393 82.4061 83.8125 0.0130584 0.0121297 2024-03-04T16:22:18.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:18.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:18.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:19.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:19.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:19.162 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:19.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:19.264 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.300 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:19.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:19.401 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:19.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:19.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:19.586 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:19.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:19.731 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:19.833 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:19.863 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:19.875 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 43 16 56645 56629 82.2852 77.25 0.0155166 0.0121462 2024-03-04T16:22:19.915 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:19.965 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:19.997 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:19.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:20.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:20.139 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:20.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:20.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:20.382 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:20.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:20.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:20.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:20.662 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:20.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:22:20.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:20.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:20.783 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T16:22:20.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:20.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:20.877 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 44 16 57911 57895 82.2132 79.125 0.00939535 0.0121566 2024-03-04T16:22:20.950 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:20.978 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:20.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:21.080 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:20 smithi176 ceph-mon[33619]: pgmap v921: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:22:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3278341364' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:20 smithi176 ceph-mon[33619]: from='client.25822 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.112 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:21.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:21.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[29779]: pgmap v921: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:22:21.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3278341364' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[29779]: from='client.25822 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[38113]: pgmap v921: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-03-04T16:22:21.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3278341364' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:20 smithi038 ceph-mon[38113]: from='client.25822 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:22:21.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:21.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:21.386 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:21.488 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.521 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:21.623 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:21.656 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:21.757 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:21.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:21.875 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 45 16 59238 59222 82.2291 82.9375 0.0208693 0.0121556 2024-03-04T16:22:21.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:21.926 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:21.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:22.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:22.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 3 1 2024-03-04T16:22:22.055 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:21 smithi038 ceph-mon[29779]: from='client.25822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:22:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:21 smithi038 ceph-mon[29779]: osdmap e430: 8 total, 8 up, 7 in 2024-03-04T16:22:22.081 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:21 smithi038 ceph-mon[38113]: from='client.25822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:22:22.081 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:21 smithi038 ceph-mon[38113]: osdmap e430: 8 total, 8 up, 7 in 2024-03-04T16:22:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:21 smithi176 ceph-mon[33619]: from='client.25822 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:22:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:21 smithi176 ceph-mon[33619]: osdmap e430: 8 total, 8 up, 7 in 2024-03-04T16:22:22.156 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:22.182 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:22.284 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.341 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:22.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:22.442 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:22.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.482 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:22.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:22.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:22.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:22.714 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:22.769 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:22.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:22.871 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:22.878 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 46 16 60421 60405 82.0487 73.9375 0.0263141 0.0121817 2024-03-04T16:22:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:22.918 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:22.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:23.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.065 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:22 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:22 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:22 smithi176 ceph-mon[33619]: pgmap v923: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:22:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:22 smithi176 ceph-mon[33619]: osdmap e431: 8 total, 8 up, 7 in 2024-03-04T16:22:23.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:23.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[29779]: pgmap v923: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[29779]: osdmap e431: 8 total, 8 up, 7 in 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[38113]: pgmap v923: 148 pgs: 148 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-04T16:22:23.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:22 smithi038 ceph-mon[38113]: osdmap e431: 8 total, 8 up, 7 in 2024-03-04T16:22:23.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.370 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:23.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:23.517 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:23.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:23.656 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:23.687 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:23.710 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:23.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:23.793 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.824 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:23.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:23.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:23.832 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:22:23.882 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 47 15 61609 61594 81.8839 74.3125 0.00949757 0.0122073 2024-03-04T16:22:23.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:23.959 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:23.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1287716586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:23.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1287716586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:23.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:23.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:24.060 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:24.089 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1287716586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:24.192 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:24.235 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:24.337 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:24.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.378 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:24.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:24.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.523 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:24.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:24.623 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:24.665 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:24.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:24.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:24.876 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 48 16 62915 62899 81.877 81.5625 0.00801128 0.0122072 2024-03-04T16:22:24.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:24.934 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:24.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:25.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:25.072 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: pgmap v925: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: osdmap e432: 8 total, 8 up, 7 in 2024-03-04T16:22:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:24 smithi176 ceph-mon[33619]: osdmap e433: 8 total, 8 up, 7 in 2024-03-04T16:22:25.174 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:25.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:25.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: pgmap v925: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: osdmap e432: 8 total, 8 up, 7 in 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[29779]: osdmap e433: 8 total, 8 up, 7 in 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: pgmap v925: 148 pgs: 148 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-04T16:22:25.193 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-04T16:22:25.193 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-04T16:22:25.193 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: osdmap e432: 8 total, 8 up, 7 in 2024-03-04T16:22:25.193 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:24 smithi038 ceph-mon[38113]: osdmap e433: 8 total, 8 up, 7 in 2024-03-04T16:22:25.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:25.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:25.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:25.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:25.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:25.483 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:25.585 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:25.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:25.718 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:25.751 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:25.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:25.853 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:25.876 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 49 16 64254 64238 81.9138 83.6875 0.00855467 0.0122022 2024-03-04T16:22:25.879 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:25.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:25.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:25.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.021 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:26.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:26.122 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:26.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:26.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:26.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:26.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:26.424 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:26.433 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:22:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:22:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:22:26.434 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:22:26.434 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:22:26.526 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:26.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:26.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:26.702 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:26 smithi038 ceph-mon[29779]: pgmap v928: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-04T16:22:26.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:26 smithi038 ceph-mon[38113]: pgmap v928: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-04T16:22:26.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:26.835 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:26.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:26 smithi176 ceph-mon[33619]: pgmap v928: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-04T16:22:26.876 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 50 16 65496 65480 81.8279 77.625 0.0100906 0.0122152 2024-03-04T16:22:26.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:26.979 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:26.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:26.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:27.081 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:27.117 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:27.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:27.219 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:27.250 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:27.352 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:27.385 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:27.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.531 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:27.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:27.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:22:27.632 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:27.678 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:27 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:27.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:27 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:27 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:27 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:27.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:27.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:27 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:27.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:27 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:27.877 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 51 16 66662 66646 81.6512 72.875 0.0241274 0.0122407 2024-03-04T16:22:27.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:27.893 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:27.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:27.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:28.020 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:28.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:28.160 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:28.262 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:28.289 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:28.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:28.423 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:28.525 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:28.568 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:28.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:28.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:28.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:28.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:28 smithi038 ceph-mon[29779]: osdmap e434: 8 total, 8 up, 7 in 2024-03-04T16:22:28.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:28 smithi038 ceph-mon[29779]: pgmap v930: 148 pgs: 1 peering, 147 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 866 B/s rd, 84 MiB/s wr, 1.34k op/s; 18 B/s, 1 objects/s recovering 2024-03-04T16:22:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:28 smithi038 ceph-mon[38113]: osdmap e434: 8 total, 8 up, 7 in 2024-03-04T16:22:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:28 smithi038 ceph-mon[38113]: pgmap v930: 148 pgs: 1 peering, 147 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 866 B/s rd, 84 MiB/s wr, 1.34k op/s; 18 B/s, 1 objects/s recovering 2024-03-04T16:22:28.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:28.845 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:28.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:28 smithi176 ceph-mon[33619]: osdmap e434: 8 total, 8 up, 7 in 2024-03-04T16:22:28.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:28 smithi176 ceph-mon[33619]: pgmap v930: 148 pgs: 1 peering, 147 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 866 B/s rd, 84 MiB/s wr, 1.34k op/s; 18 B/s, 1 objects/s recovering 2024-03-04T16:22:28.877 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 52 16 67899 67883 81.5677 77.3125 0.00367134 0.0122476 2024-03-04T16:22:28.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:28.986 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:28.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:29.088 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:29.116 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:29.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:29.255 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:29.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:29.383 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:29.485 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:29.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:29.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:29.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:29.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:29.789 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:29.877 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 53 16 69064 69048 81.4024 72.8125 0.0107968 0.0122792 2024-03-04T16:22:29.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:29.920 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:29.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:30.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:30.063 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:30.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:30.305 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:30.336 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:30.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:30.469 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:30.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:30.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:30.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:30.730 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:30.832 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:30.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:30.877 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 54 16 70282 70266 81.3046 76.125 0.0395238 0.0122913 2024-03-04T16:22:30.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:30.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:30.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:31.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:31.127 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:31.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:31.261 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:31.363 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.407 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:31.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:31.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:31.558 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:31.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[29779]: pgmap v931: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 88 MiB/s wr, 1.41k op/s; 36 B/s, 3 objects/s recovering 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[29779]: osdmap e435: 8 total, 8 up, 7 in 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[38113]: pgmap v931: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 88 MiB/s wr, 1.41k op/s; 36 B/s, 3 objects/s recovering 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:31.721 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:31.722 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:31 smithi038 ceph-mon[38113]: osdmap e435: 8 total, 8 up, 7 in 2024-03-04T16:22:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:31.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:31.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:31.855 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:31.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:31 smithi176 ceph-mon[33619]: pgmap v931: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 88 MiB/s wr, 1.41k op/s; 36 B/s, 3 objects/s recovering 2024-03-04T16:22:31.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:31 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:31.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:31 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:31.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:31 smithi176 ceph-mon[33619]: osdmap e435: 8 total, 8 up, 7 in 2024-03-04T16:22:31.880 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 55 16 71485 71469 81.1895 75.1875 0.0305307 0.0123106 2024-03-04T16:22:31.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:31.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:31.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:32.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:32.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:32.244 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:32.284 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:32.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:32.457 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:22:32.528 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:32.598 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:32.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:32.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:32.845 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:32.873 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:32.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:32.880 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 56 16 72549 72533 80.9271 66.5 0.00678206 0.0123509 2024-03-04T16:22:32.952 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:32.975 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:33.003 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:33.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:33.157 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:33.259 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:33.300 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:22:33.303 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:33.346 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:22:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:22:33.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:33.454 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:22:33.455 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T16:22:33.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:33.497 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:22:33.539 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:22:33.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:33.584 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[29779]: pgmap v933: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 593 B/s rd, 56 MiB/s wr, 902 op/s; 31 B/s, 2 objects/s recovering 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3205928815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[29779]: from='client.35906 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3608825151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[38113]: pgmap v933: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 593 B/s rd, 56 MiB/s wr, 902 op/s; 31 B/s, 2 objects/s recovering 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3205928815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[38113]: from='client.35906 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.585 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3608825151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:33.586 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.618 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:33.637 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:22:33.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:22:33.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:22:33.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:22:33.654 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:22:33.655 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:22:33.655 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:22:33.655 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:22:33.655 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:22:33.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:33.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:33.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:33 smithi176 ceph-mon[33619]: pgmap v933: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 593 B/s rd, 56 MiB/s wr, 902 op/s; 31 B/s, 2 objects/s recovering 2024-03-04T16:22:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3205928815' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:33 smithi176 ceph-mon[33619]: from='client.35906 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:22:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:33 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3608825151' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:33.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:33.880 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 57 16 73524 73508 80.5764 60.9375 0.00876795 0.0124045 2024-03-04T16:22:33.904 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:33.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:34.006 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:34.046 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:34.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.179 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:34.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:34.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:34.318 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:34.420 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:34.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:34.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:34.553 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.597 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:34.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:34.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:34.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[29779]: from='client.35906 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:22:34.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[29779]: osdmap e436: 8 total, 8 up, 7 in 2024-03-04T16:22:34.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:34.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[29779]: pgmap v935: 148 pgs: 148 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 35 B/s, 3 objects/s recovering 2024-03-04T16:22:34.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[38113]: from='client.35906 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:22:34.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[38113]: osdmap e436: 8 total, 8 up, 7 in 2024-03-04T16:22:34.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:34.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:34 smithi038 ceph-mon[38113]: pgmap v935: 148 pgs: 148 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 35 B/s, 3 objects/s recovering 2024-03-04T16:22:34.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:34.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:34.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:34 smithi176 ceph-mon[33619]: from='client.35906 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:22:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:34 smithi176 ceph-mon[33619]: osdmap e436: 8 total, 8 up, 7 in 2024-03-04T16:22:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:34 smithi176 ceph-mon[33619]: pgmap v935: 148 pgs: 148 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 99 MiB/s wr, 1.59k op/s; 35 B/s, 3 objects/s recovering 2024-03-04T16:22:34.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 58 16 74730 74714 80.4855 75.375 0.0139305 0.0124191 2024-03-04T16:22:34.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:34.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:34.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:35.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:35.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:35.147 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:35.176 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:35.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:35.318 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:35.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:35.445 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:35.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:35.587 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:35.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:35.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:35.723 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:35.826 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:35.857 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:35.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:35.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:35.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 59 16 75850 75834 80.3077 70 0.0276309 0.0124464 2024-03-04T16:22:35.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:35.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:36.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:36.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:36.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:36.271 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:36.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:36.409 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:36.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:36.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[29779]: osdmap e437: 8 total, 8 up, 7 in 2024-03-04T16:22:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:36.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:36 smithi038 ceph-mon[38113]: osdmap e437: 8 total, 8 up, 7 in 2024-03-04T16:22:36.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:36.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:36.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:36.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:36.791 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:36.835 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:36.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:36 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:36.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:36 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:36 smithi176 ceph-mon[33619]: osdmap e437: 8 total, 8 up, 7 in 2024-03-04T16:22:36.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:22:36.882128+0000 min lat: 0.00129433 max lat: 1.98384 avg lat: 0.0124615 2024-03-04T16:22:36.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:22:36.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 60 15 77043 77028 80.2129 74.625 0.00648358 0.0124615 2024-03-04T16:22:36.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:36.970 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:36.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:37.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:37.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:37.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:37.253 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:37.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:37.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:37.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:37.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:37.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:37.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:37 smithi038 ceph-mon[29779]: pgmap v937: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-04T16:22:37.532 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:37 smithi038 ceph-mon[38113]: pgmap v937: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-04T16:22:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.542 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:37.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:37.643 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:37.675 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:37.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:37.777 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:37.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:37 smithi176 ceph-mon[33619]: pgmap v937: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-04T16:22:37.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 61 15 78135 78120 80.0167 68.25 0.0139163 0.0124916 2024-03-04T16:22:37.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:37.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 5 1 2024-03-04T16:22:37.937 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:37.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:37.976 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:38.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:38.102 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:38.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:38.204 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.263 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:38.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:38.364 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:38.394 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:38.441 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:22:38.495 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:38.545 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:38.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:38.647 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:38.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[29779]: pgmap v938: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 700 B/s rd, 62 MiB/s wr, 991 op/s 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4078584016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[29779]: from='client.25840 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[38113]: pgmap v938: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 700 B/s rd, 62 MiB/s wr, 991 op/s 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4078584016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:38 smithi038 ceph-mon[38113]: from='client.25840 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.822 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:38.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:38 smithi176 ceph-mon[33619]: pgmap v938: 148 pgs: 148 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 700 B/s rd, 62 MiB/s wr, 991 op/s 2024-03-04T16:22:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4078584016' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:38 smithi176 ceph-mon[33619]: from='client.25840 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:22:38.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 62 16 79157 79141 79.7553 63.8125 0.0172529 0.0125331 2024-03-04T16:22:38.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:38.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:38.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:39.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:22:39.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:39.100 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:39.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.250 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:39.351 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.386 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:39.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:39.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:39.525 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:39.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:39.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:39.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:39.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[29779]: from='client.25840 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[29779]: osdmap e438: 8 total, 8 up, 7 in 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[38113]: from='client.25840 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[38113]: osdmap e438: 8 total, 8 up, 7 in 2024-03-04T16:22:39.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:39 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:39.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:39.803 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:39 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:22:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:39 smithi176 ceph-mon[33619]: from='client.25840 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:22:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:39 smithi176 ceph-mon[33619]: osdmap e438: 8 total, 8 up, 7 in 2024-03-04T16:22:39.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:39 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:39.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 63 16 80168 80152 79.4923 63.1875 0.00639835 0.0125731 2024-03-04T16:22:39.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:39.942 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:39.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:40.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:40.071 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:40.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:40.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:40.315 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:40.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:40.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:40.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:40.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:40.629 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:40.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:40.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:40.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[29779]: pgmap v940: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 813 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:22:40.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[29779]: 11.0 scrub starts 2024-03-04T16:22:40.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[29779]: osdmap e439: 8 total, 8 up, 7 in 2024-03-04T16:22:40.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[38113]: pgmap v940: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 813 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:22:40.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[38113]: 11.0 scrub starts 2024-03-04T16:22:40.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:40 smithi038 ceph-mon[38113]: osdmap e439: 8 total, 8 up, 7 in 2024-03-04T16:22:40.781 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:40.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:40 smithi176 ceph-mon[33619]: pgmap v940: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 813 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-04T16:22:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:40 smithi176 ceph-mon[33619]: 11.0 scrub starts 2024-03-04T16:22:40.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:40 smithi176 ceph-mon[33619]: osdmap e439: 8 total, 8 up, 7 in 2024-03-04T16:22:40.881 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 64 16 80929 80913 78.9934 47.5625 0.00338661 0.0126454 2024-03-04T16:22:40.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:40.910 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:40.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:41.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:41.041 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:41.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:41.172 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:41.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:41.319 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:41.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.466 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:41.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:41.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:41.595 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:41.685 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:41.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:41.718 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:41 smithi038 ceph-mon[29779]: 11.b scrub starts 2024-03-04T16:22:41.718 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:41 smithi038 ceph-mon[38113]: 11.b scrub starts 2024-03-04T16:22:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.781 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:41.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:41 smithi176 ceph-mon[33619]: 11.b scrub starts 2024-03-04T16:22:41.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:41.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 65 16 81872 81856 78.6834 58.9375 0.0104294 0.0127033 2024-03-04T16:22:41.912 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:41.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:42.014 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:42.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:42.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:42.142 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:42.165 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:42.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:42.305 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:42.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:42.439 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:42.541 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:42.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:42.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:42.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[29779]: pgmap v942: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 811 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[29779]: 11.0 scrub ok 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[38113]: pgmap v942: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 811 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:22:42.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:42 smithi038 ceph-mon[38113]: 11.0 scrub ok 2024-03-04T16:22:42.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:42.836 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:42.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:42 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:22:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:42 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:22:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:42 smithi176 ceph-mon[33619]: pgmap v942: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 811 B/s rd, 55 MiB/s wr, 874 op/s 2024-03-04T16:22:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:42 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:22:42.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:42 smithi176 ceph-mon[33619]: 11.0 scrub ok 2024-03-04T16:22:42.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 66 16 82819 82803 78.388 59.1875 0.0105401 0.0127516 2024-03-04T16:22:42.942 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:42.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:43.085 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:43.114 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:43.218 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:43.245 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:43.347 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:43.378 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:43.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:43.512 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:43.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.529 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:43.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:22:43.601 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 11.b scrub ok 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:22:43.602 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 11.b scrub ok 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:22:43.603 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:43 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:22:43.631 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:43.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:43.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:43.790 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:43.818 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:43.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:43.859 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 11.b scrub ok 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:22:43.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:43 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:22:43.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 67 16 83915 83899 78.2404 68.5 0.0117409 0.0127754 2024-03-04T16:22:43.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:43.953 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:43.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:44.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:44.082 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:44.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:44.223 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:44.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:44.375 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:44.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:44.503 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:44.605 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:44.644 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:44.746 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:44.788 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:44.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: pgmap v943: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:44.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:44 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:22:44.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 68 16 84932 84916 78.0245 63.5625 0.0137597 0.0128119 2024-03-04T16:22:44.890 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:44.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:44.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:22:44.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:22:44.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: pgmap v943: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: pgmap v943: 148 pgs: 148 active+clean; 5.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:44.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:44 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:22:44.926 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:44.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:45.028 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:45.056 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:45.158 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:45.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:45.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:45.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:45.323 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:45.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:45.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.470 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:45.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:45.500 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:22:45.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:45.594 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:45.690 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:22:45.690 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:22:45.697 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:45.723 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:22:45.725 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:22:45.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:22:45.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:22:45.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:22:45.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:45 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:22:45.825 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:22:45.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:45 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:22:45.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 69 16 85863 85847 77.737 58.1875 0.0250487 0.0128578 2024-03-04T16:22:45.896 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:45.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:45.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.064 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:46.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:46.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:46.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:46.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:46.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:46.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:46.492 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:46.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:46.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:46.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:46.737 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.766 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:46.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:22:46.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:22:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: pgmap v944: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-04T16:22:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:22:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:22:46.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:46 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:46.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:46.883 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 70 16 86834 86818 77.4932 60.6875 0.0189315 0.0128988 2024-03-04T16:22:46.910 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:46.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: pgmap v944: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-04T16:22:46.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: pgmap v944: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:22:46.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:22:46.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:46 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:22:47.012 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:47.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:47.117 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:22:47.141 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:47.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:47.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.346 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:47.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:47.447 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:47.477 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:47.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:47.642 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:47.744 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:47.774 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:47.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 6.d scrub starts 2024-03-04T16:22:47.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 6.d scrub ok 2024-03-04T16:22:47.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:22:47.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:22:47.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:22:47.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:22:47.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:22:47.820 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:22:47.820 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 6.d scrub starts 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 6.d scrub ok 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:22:47.821 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:47 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:22:47.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 6.d scrub starts 2024-03-04T16:22:47.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 6.d scrub ok 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:22:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:47 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:22:47.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:47.885 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 71 16 87893 87877 77.3316 66.1875 0.0118242 0.0129252 2024-03-04T16:22:47.913 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:47.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:48.015 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:48.045 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:48.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:48.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.187 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:48.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:48.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:48.315 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:48.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.468 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:48.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:48.569 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:48.612 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:48.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:48.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:48.742 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:48.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:48 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:22:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:48 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:22:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:48 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:22:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:48 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:22:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:48 smithi176 ceph-mon[33619]: pgmap v945: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 551 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-04T16:22:48.884 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:48.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:48.886 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 72 16 88904 88888 77.1349 63.1875 0.0207428 0.0129589 2024-03-04T16:22:48.986 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[29779]: pgmap v945: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 551 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:22:48.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:48 smithi038 ceph-mon[38113]: pgmap v945: 148 pgs: 148 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 551 B/s rd, 54 MiB/s wr, 868 op/s 2024-03-04T16:22:49.035 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:49.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:49.174 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:49.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:49.305 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:49.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:49.453 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:49.547 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-04T16:22:49.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:49.603 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:49.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:49.732 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:49.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:22:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:49 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:22:49.886 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 73 16 89993 89977 77.0105 68.0625 0.0132292 0.0129799 2024-03-04T16:22:49.901 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:49.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:22:49.939 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:22:49.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:22:49.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:22:49.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:22:49.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:49 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:22:50.003 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:50.033 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:50.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:50.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:50.181 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:50.283 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.320 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:50.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:50.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:50.476 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:50.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:50.564 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: osdmap e440: 8 total, 8 up, 7 in 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: pgmap v947: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: osdmap e441: 8 total, 8 up, 7 in 2024-03-04T16:22:50.566 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2455492118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: osdmap e440: 8 total, 8 up, 7 in 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: pgmap v947: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: osdmap e441: 8 total, 8 up, 7 in 2024-03-04T16:22:50.567 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2455492118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:50.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:50.608 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:50.613 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:22:50.657 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:22:50.702 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:22:50.709 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:50.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:50.745 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:22:50.787 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:22:50.827 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:22:50.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2335465742' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: osdmap e440: 8 total, 8 up, 7 in 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: pgmap v947: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: osdmap e441: 8 total, 8 up, 7 in 2024-03-04T16:22:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2455492118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:22:50.872 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:22:50.883 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:22:50.884 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:22:50.886 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 74 16 90942 90926 76.7713 59.3125 0.0118827 0.0130198 2024-03-04T16:22:50.889 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:50.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:50.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.038 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:51.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:51.071 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 7 1 2024-03-04T16:22:51.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:51.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:51.292 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:51.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:51.341 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.376 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:51.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:51.477 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:51.508 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:51.610 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:51.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:22:51.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:22:51.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:22:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:22:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:22:51.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:51 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:22:51.758 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:51.788 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:22:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:51 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:22:51.888 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 75 16 91932 91916 76.5706 61.875 0.0173082 0.0130529 2024-03-04T16:22:51.890 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:51.918 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:51.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:52.020 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:52.061 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:52.163 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:52.195 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:52.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:52.297 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:52.326 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:52.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:52.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:52.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:52.613 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:52.715 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:52.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:22:52.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:22:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:22:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:22:52.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[29779]: pgmap v949: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-04T16:22:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:22:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:22:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:22:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:22:52.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:52 smithi038 ceph-mon[38113]: pgmap v949: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-04T16:22:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:52.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:52.846 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:52.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:52 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:22:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:52 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:22:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:52 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:22:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:52 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:22:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:52 smithi176 ceph-mon[33619]: pgmap v949: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 857 op/s 2024-03-04T16:22:52.875 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:52.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:52.888 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 76 16 92988 92972 76.4315 66 0.0167289 0.013078 2024-03-04T16:22:52.976 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:53.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:53.155 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:53.257 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:53.284 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:53.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:53.386 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:53.411 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:53.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:22:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:22:53.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.545 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:53.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:53.646 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.696 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:53.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:53.797 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.842 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:53.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:53.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:22:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:22:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:22:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 11.d scrub starts 2024-03-04T16:22:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:22:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:22:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:22:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:22:53.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:53.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:22:53.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 11.d scrub starts 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:22:53.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:53.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:22:53.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:22:53.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 11.d scrub starts 2024-03-04T16:22:53.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:22:53.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:22:53.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:22:53.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:22:53.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:22:53.890 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 77 16 93935 93919 76.2057 59.1875 0.0334867 0.013116 2024-03-04T16:22:53.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:53.986 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:53.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:54.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:54.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.141 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:54.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:54.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:54.269 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:54.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:54.371 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:54.410 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:54.512 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:54.539 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:22:54.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:54.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:54.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:54.831 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:54.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:54.891 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 78 16 94862 94846 75.9715 57.9375 0.011208 0.0131576 2024-03-04T16:22:54.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: pgmap v950: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 20 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1131989085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[29779]: from='client.35930 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:54.940 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: pgmap v950: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 20 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1131989085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:54.941 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:54 smithi038 ceph-mon[38113]: from='client.35930 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:54.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:55.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:55.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: pgmap v950: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 20 KiB/s rd, 64 MiB/s wr, 1.06k op/s 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1131989085' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:55.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:54 smithi176 ceph-mon[33619]: from='client.35930 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:22:55.146 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:55.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:55.279 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:55.380 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.426 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:55.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:55.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:55.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:55.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.702 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:55.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:55.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.833 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:55.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:55.890 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 79 16 95903 95887 75.8333 65.0625 0.0156659 0.0131812 2024-03-04T16:22:55.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:55.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:55.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:55.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:22:55.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: from='client.35930 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: osdmap e442: 8 total, 8 up, 7 in 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: 11.d scrub ok 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:22:55.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[29779]: osdmap e443: 8 total, 8 up, 7 in 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: from='client.35930 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: osdmap e442: 8 total, 8 up, 7 in 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: 11.d scrub ok 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:22:55.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:55 smithi038 ceph-mon[38113]: osdmap e443: 8 total, 8 up, 7 in 2024-03-04T16:22:56.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:56.097 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: from='client.35930 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: osdmap e442: 8 total, 8 up, 7 in 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: 11.d scrub ok 2024-03-04T16:22:56.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:22:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:22:56.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:55 smithi176 ceph-mon[33619]: osdmap e443: 8 total, 8 up, 7 in 2024-03-04T16:22:56.199 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.243 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:56.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:56.344 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:56.371 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:56.473 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:56.501 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:56.603 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:56.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:56.684 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:22:56.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[29779]: 11.2 scrub starts 2024-03-04T16:22:56.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[29779]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 KiB/s rd, 52 MiB/s wr, 932 op/s 2024-03-04T16:22:56.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:22:56.696 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:22:56.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[38113]: 11.2 scrub starts 2024-03-04T16:22:56.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[38113]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 KiB/s rd, 52 MiB/s wr, 932 op/s 2024-03-04T16:22:56.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:22:56.697 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:56 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:22:56.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:56.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:56.890 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:2024-03-04T16:22:56.891703+0000 min lat: 0.00129433 max lat: 1.98384 avg lat: 0.0132055 2024-03-04T16:22:56.890 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-04T16:22:56.891 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 80 16 96936 96920 75.6923 64.5625 0.0282207 0.0132055 2024-03-04T16:22:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:56.921 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:56.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:56.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:22:57.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:97052 11.2 11:48787bba:::benchmark_data_smithi038_182448_object97051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:22:57.029603+0000", 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.011398080999999999, 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011434110000000001, 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:22:57.044 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 97052 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029603+0000", 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.045 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029603+0000", 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029605+0000", 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.086e-06 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.046 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029632+0000", 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6597e-05 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029643+0000", 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1841999999999999e-05 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:22:57.047 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.029652+0000", 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.2269999999999993e-06 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.030828+0000", 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0011764970000000001 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:22:57.048 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.030966+0000", 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013767900000000001 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.031111+0000", 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014557099999999999 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.034657+0000", 2024-03-04T16:22:57.049 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0035459160000000001 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:97060 11.2 11:43726d1e:::benchmark_data_smithi038_182448_object97059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:22:57.040190+0000", 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.00081107800000000002, 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094436699999999999, 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:22:57.050 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 97060 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040190+0000", 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.051 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040190+0000", 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040192+0000", 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7239999999999999e-06 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:22:57.052 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040218+0000", 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6648e-05 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040233+0000", 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4470999999999999e-05 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040241+0000", 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.5990000000000002e-06 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.053 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040349+0000", 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107817 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040486+0000", 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000136809 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.054 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 0,3", 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040661+0000", 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000175217 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.16017.0:97061 11.2 11.33bb0422 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e443)", 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:22:57.040262+0000", 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 0.000739023, 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00094282899999999995, 2024-03-04T16:22:57.055 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "queued for pg", 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.16017", 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 97061 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040262+0000", 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040262+0000", 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040263+0000", 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8800000000000001e-07 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.057 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040288+0000", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.5055e-05 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040296+0000", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0870000000000007e-06 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:22:57.040303+0000", 2024-03-04T16:22:57.058 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.7560000000000002e-06 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout: "num_ops": 3 2024-03-04T16:22:57.059 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:22:57.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:56 smithi176 ceph-mon[33619]: 11.2 scrub starts 2024-03-04T16:22:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:56 smithi176 ceph-mon[33619]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 60 KiB/s rd, 52 MiB/s wr, 932 op/s 2024-03-04T16:22:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:56 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:22:57.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:56 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:22:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:57.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:57.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.322 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:57.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:57.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:57.452 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:57.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:57.595 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:57.697 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:22:57.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:22:57.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:57.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:22:57.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:22:57.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:22:57.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:22:57.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:22:57.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:22:57.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:22:57.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:22:57.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:22:57.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:57 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:22:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:57.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:57.886 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:57.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:57.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:57.890 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 81 16 97886 97870 75.4908 59.375 0.00742535 0.0132415 2024-03-04T16:22:57.988 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:22:58.022 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:22:58.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:57 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:22:58.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:58.153 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:58.255 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:22:58.291 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:58.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:22:58.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:58.425 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:58.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:58.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:58.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:58.699 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:22:58.705 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:58.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:22:58.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:22:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.836 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:22:58.893 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 82 16 98968 98952 75.3927 67.625 0.0248313 0.0132584 2024-03-04T16:22:58.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:22:58.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:58.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 KiB/s rd, 53 MiB/s wr, 939 op/s 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 11.2 scrub ok 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:22:58.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 KiB/s rd, 53 MiB/s wr, 939 op/s 2024-03-04T16:22:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:22:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:22:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 11.2 scrub ok 2024-03-04T16:22:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:22:58.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:22:58 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:22:59.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:59.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:59.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 KiB/s rd, 53 MiB/s wr, 939 op/s 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 11.2 scrub ok 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:22:59.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:22:58 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:22:59.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:22:59.244 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:22:59.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.385 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:59.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:22:59.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.532 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:59.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:59.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:22:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.703 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:59.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:22:59.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:22:59.834 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:22:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:22:59.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 83 16 100028 100012 75.2819 66.25 0.025811 0.0132781 2024-03-04T16:22:59.936 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:22:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:22:59.987 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:22:59.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:00.033 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:00.089 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.139 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:00.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:00.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:00.281 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:00.289 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:00.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.438 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:00.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:00.539 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:00.564 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:00.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:00.693 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:00.701 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:00.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:00.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:00.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 84 16 101011 100995 75.1171 61.4375 0.00653603 0.0133078 2024-03-04T16:23:00.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:00.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[29779]: 6.1e scrub starts 2024-03-04T16:23:00.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[29779]: 6.1e scrub ok 2024-03-04T16:23:00.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[29779]: pgmap v955: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 KiB/s rd, 78 MiB/s wr, 1.34k op/s 2024-03-04T16:23:00.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[38113]: 6.1e scrub starts 2024-03-04T16:23:00.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[38113]: 6.1e scrub ok 2024-03-04T16:23:00.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:00 smithi038 ceph-mon[38113]: pgmap v955: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 KiB/s rd, 78 MiB/s wr, 1.34k op/s 2024-03-04T16:23:00.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:00.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:01.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:01.082 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:01.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:00 smithi176 ceph-mon[33619]: 6.1e scrub starts 2024-03-04T16:23:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:00 smithi176 ceph-mon[33619]: 6.1e scrub ok 2024-03-04T16:23:01.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:00 smithi176 ceph-mon[33619]: pgmap v955: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 55 KiB/s rd, 78 MiB/s wr, 1.34k op/s 2024-03-04T16:23:01.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:01.228 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:01.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:01.374 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:01.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:01.516 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:01.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:01.662 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:01.764 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:01.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:01.817 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-04T16:23:01.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 85 16 102140 102124 75.0634 70.5625 0.00922037 0.0133162 2024-03-04T16:23:01.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:01.945 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:01.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:02.046 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:02.086 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:02.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:23:02.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.1b deep-scrub starts 2024-03-04T16:23:02.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:01 smithi176 ceph-mon[33619]: 4.1b deep-scrub ok 2024-03-04T16:23:02.188 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:02.218 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:02.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.1b deep-scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[29779]: 4.1b deep-scrub ok 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:23:02.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:23:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:23:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:23:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:23:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.1b deep-scrub starts 2024-03-04T16:23:02.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:01 smithi038 ceph-mon[38113]: 4.1b deep-scrub ok 2024-03-04T16:23:02.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:02.353 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:02.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:02.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:02.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.619 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:02.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:02.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:02.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:02.851 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:02.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:02.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:02.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:02.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 86 16 103301 103285 75.0343 72.5625 0.00689704 0.0133223 2024-03-04T16:23:02.982 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:03.019 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[29779]: pgmap v956: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 35 KiB/s rd, 53 MiB/s wr, 914 op/s 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[29779]: osdmap e444: 8 total, 8 up, 7 in 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[38113]: pgmap v956: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 35 KiB/s rd, 53 MiB/s wr, 914 op/s 2024-03-04T16:23:03.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:23:03.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:23:03.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:23:03.068 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:02 smithi038 ceph-mon[38113]: osdmap e444: 8 total, 8 up, 7 in 2024-03-04T16:23:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:02 smithi176 ceph-mon[33619]: pgmap v956: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 35 KiB/s rd, 53 MiB/s wr, 914 op/s 2024-03-04T16:23:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:02 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:23:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:02 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:23:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:02 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2259567332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:23:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:02 smithi176 ceph-mon[33619]: osdmap e444: 8 total, 8 up, 7 in 2024-03-04T16:23:03.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:03.148 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:03.250 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:03.292 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:03.394 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:03.434 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:03.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:03.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:03.580 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:03.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:03.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:03.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:03.827 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:03.859 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:03.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 87 16 104275 104259 74.8715 60.875 0.0176396 0.0133513 2024-03-04T16:23:03.961 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:03.996 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:03.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:03.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:04.099 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:23:04.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:03 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:23:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.124 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:04.225 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:23:04.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:23:04.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:23:04.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:23:04.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:23:04.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:23:04.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:03 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:23:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:04.368 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:04.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:04.540 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:04.642 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.680 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:04.781 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:04.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:04.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:04.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 88 16 105140 105124 74.635 54.0625 0.0068312 0.0133936 2024-03-04T16:23:04.916 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:04.949 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:04.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:23:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: 11.6 scrub starts 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: pgmap v958: 148 pgs: 148 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 9.9 KiB/s rd, 63 MiB/s wr, 1.03k op/s 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[29779]: osdmap e445: 8 total, 8 up, 7 in 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:23:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: 11.6 scrub starts 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: pgmap v958: 148 pgs: 148 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 9.9 KiB/s rd, 63 MiB/s wr, 1.03k op/s 2024-03-04T16:23:04.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:04 smithi038 ceph-mon[38113]: osdmap e445: 8 total, 8 up, 7 in 2024-03-04T16:23:05.049 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.096 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:05.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: 11.6 scrub starts 2024-03-04T16:23:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: pgmap v958: 148 pgs: 148 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 9.9 KiB/s rd, 63 MiB/s wr, 1.03k op/s 2024-03-04T16:23:05.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:04 smithi176 ceph-mon[33619]: osdmap e445: 8 total, 8 up, 7 in 2024-03-04T16:23:05.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:05.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:05.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:05.358 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:05.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:05.511 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:05.613 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:05.646 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:05.748 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.781 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:05.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:05.883 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:05.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout: 89 16 106144 106128 74.5015 62.75 0.01189 0.0134178 2024-03-04T16:23:05.909 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:05.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:05.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:06.011 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:06.050 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:05 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:23:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:05 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:23:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:05 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:23:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:05 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:23:06.131 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:23:06.132 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:05 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:23:06.153 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:06.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:06.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:06.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:06.419 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.470 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:06.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:06.571 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:06.611 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:06.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:06.713 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:06.740 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:06.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:06.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:06.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:06.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total time run: 90.0071 2024-03-04T16:23:06.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Total writes made: 107136 2024-03-04T16:23:06.894 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Write size: 65536 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Object size: 65536 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Bandwidth (MB/sec): 74.3941 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Bandwidth: 28.3367 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max bandwidth (MB/sec): 153.625 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min bandwidth (MB/sec): 0 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average IOPS: 1190 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev IOPS: 453.388 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max IOPS: 2458 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min IOPS: 0 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Average Latency(s): 0.0134374 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Stddev Latency(s): 0.0398407 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Max latency(s): 1.98384 2024-03-04T16:23:06.895 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Min latency(s): 0.00129433 2024-03-04T16:23:06.900 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Cleaning up (deleting benchmark objects) 2024-03-04T16:23:06.974 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:07.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:06 smithi176 ceph-mon[33619]: pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 5.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-04T16:23:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:06 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:23:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:06 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:23:07.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:07.149 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:07.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[29779]: pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 5.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-04T16:23:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:23:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:23:07.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[38113]: pgmap v960: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 5.7 KiB/s rd, 71 MiB/s wr, 1.15k op/s 2024-03-04T16:23:07.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:23:07.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:06 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:23:07.251 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:07.280 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:07.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:07.429 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:07.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:07.531 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:07.558 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:07.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:07.699 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:07.801 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:07.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:07.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:07.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 2 1 2024-03-04T16:23:07.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:07.960 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:07.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:07.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:08.055 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:23:08.055 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[38113]: 11.6 scrub ok 2024-03-04T16:23:08.055 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:23:08.056 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:23:08.056 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[29779]: 11.6 scrub ok 2024-03-04T16:23:08.056 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:07 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:23:08.061 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:08.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:07 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:23:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:07 smithi176 ceph-mon[33619]: 11.6 scrub ok 2024-03-04T16:23:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:07 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:23:08.112 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:08.214 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:08.248 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:08.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:08.350 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:08.432 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:08.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:08.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:08.562 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:08.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:08.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:08.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:08.832 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:08.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:08.971 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:08.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:08.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:09.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:09.111 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:09.213 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.241 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:09.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:09.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:23:09.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:23:09.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 819 op/s 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 819 op/s 2024-03-04T16:23:09.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:23:09.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:23:09.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:09.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:09.342 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:09.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: pgmap v961: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 50 MiB/s wr, 819 op/s 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:09.382 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:09.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:09.628 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:09.657 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:09.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:09.786 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:09.888 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:09.931 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:09.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:09.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 4.7 scrub starts 2024-03-04T16:23:09.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 4.7 scrub ok 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 3.1e deep-scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 3.1e deep-scrub ok 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 4.7 scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 4.7 scrub ok 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 3.1e deep-scrub starts 2024-03-04T16:23:09.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 3.1e deep-scrub ok 2024-03-04T16:23:09.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:09 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:23:10.033 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:10.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:10.159 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:10.187 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:10.288 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.320 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:10.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:10.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 4.7 scrub starts 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 4.7 scrub ok 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 3.1e deep-scrub starts 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 3.1e deep-scrub ok 2024-03-04T16:23:10.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:09 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:23:10.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:10.462 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:10.490 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:23:10.490 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [], 2024-03-04T16:23:10.490 INFO:teuthology.orchestra.run.smithi038.stdout: "complaint_time": 30, 2024-03-04T16:23:10.490 INFO:teuthology.orchestra.run.smithi038.stdout: "num_blocked_ops": 0 2024-03-04T16:23:10.490 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:23:10.564 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:10.606 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:10.708 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:10.734 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:10.836 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:10.885 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:10.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:10.986 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:11.015 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:11.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 11.e scrub starts 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: pgmap v962: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 68 MiB/s wr, 1.31k op/s 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:23:11.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 11.e scrub starts 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: pgmap v962: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 68 MiB/s wr, 1.31k op/s 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:23:11.123 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:10 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:23:11.145 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:11.246 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:11.260 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:23:11.290 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:11.306 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:23:11.355 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:23:11.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 11.e scrub starts 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: pgmap v962: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.4 KiB/s rd, 68 MiB/s wr, 1.31k op/s 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:23:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:10 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:23:11.384 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:11.391 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:11.409 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:23:11.417 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:11.448 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:23:11.486 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:23:11.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:11.535 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:23:11.549 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:11.577 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:23:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:23:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:23:11.593 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:23:11.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:11.693 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:11.795 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:11.822 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:11.924 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:11.952 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:11.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:11.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:23:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:23:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4204048026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:23:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:23:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4204048026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:12.054 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:12.080 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:12.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:12.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:23:12.256 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:11 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:23:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:11 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:23:12.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4204048026' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:12.358 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:12.384 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:12.486 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:12.517 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:12.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:12.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.651 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:12.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:12.751 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:12.891 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:12.918 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:12.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:12.919 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:23:13.019 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:13.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:13.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:13.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: 11.c scrub starts 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: pgmap v963: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.05k op/s 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:23:13.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: 11.c scrub starts 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: pgmap v963: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.05k op/s 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:23:13.165 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:12 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:23:13.180 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:13.282 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:13.313 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:23:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: 11.c scrub starts 2024-03-04T16:23:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:23:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: pgmap v963: 148 pgs: 148 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.05k op/s 2024-03-04T16:23:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:23:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:23:13.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:12 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:23:13.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:13.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:13.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:13.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:13.589 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:13.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:13.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.734 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:13.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:13.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:13.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:13.878 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:13.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:13.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:14.007 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:14.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:14.109 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:14.140 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:14.242 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:23:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:23:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: 11.e scrub ok 2024-03-04T16:23:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:14.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[29779]: osdmap e446: 8 total, 8 up, 7 in 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: 11.e scrub ok 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:23:14.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:13 smithi038 ceph-mon[38113]: osdmap e446: 8 total, 8 up, 7 in 2024-03-04T16:23:14.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:14.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:23:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:23:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: 11.e scrub ok 2024-03-04T16:23:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2847010096' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:23:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:13 smithi176 ceph-mon[33619]: osdmap e446: 8 total, 8 up, 7 in 2024-03-04T16:23:14.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:14.412 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:14.514 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:14.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:14.644 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.676 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:14.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:14.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:14.836 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:14.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:14.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:23:14.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:14.999 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:15.000 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:14 smithi038 ceph-mon[38113]: pgmap v965: 148 pgs: 148 active+clean; 6.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 1.03k op/s 2024-03-04T16:23:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:14 smithi038 ceph-mon[29779]: pgmap v965: 148 pgs: 148 active+clean; 6.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 1.03k op/s 2024-03-04T16:23:15.018 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:15.029 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:15.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:15.131 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:15.156 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:15.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:15.285 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:15.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:14 smithi176 ceph-mon[33619]: pgmap v965: 148 pgs: 148 active+clean; 6.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 34 MiB/s wr, 1.03k op/s 2024-03-04T16:23:15.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.430 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:15.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:15.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:15.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:15.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:15.710 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:15.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:15.812 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:15.855 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:15.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:15.998 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:15.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:16.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:16.128 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:16.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:16.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.272 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:16.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:16.373 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:16.404 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:16.462 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T16:23:16.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:16.534 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:16.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:16.637 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.669 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:16.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:16.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:16 smithi038 ceph-mon[29779]: pgmap v966: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.03k op/s 2024-03-04T16:23:16.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:23:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:16 smithi038 ceph-mon[38113]: pgmap v966: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.03k op/s 2024-03-04T16:23:16.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:23:16.769 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:16.795 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:23:16.797 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:16.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:16 smithi176 ceph-mon[33619]: pgmap v966: 148 pgs: 148 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 21 MiB/s wr, 1.03k op/s 2024-03-04T16:23:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:23:16.899 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:16.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:16.942 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:16.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:17.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:17.071 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:17.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:17.173 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:17.203 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:17.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:17.305 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:17.345 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:17.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:17.499 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:17.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:17.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:17.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[29779]: osdmap e447: 8 total, 8 up, 7 in 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:23:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:17 smithi038 ceph-mon[38113]: osdmap e447: 8 total, 8 up, 7 in 2024-03-04T16:23:17.774 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:17 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:17 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3596715082' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:23:17.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:17 smithi176 ceph-mon[33619]: osdmap e447: 8 total, 8 up, 7 in 2024-03-04T16:23:17.876 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:17.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:17.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:17.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:18.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:18.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:18.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:18.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:18.222 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:18.287 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:18.318 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:23:18.318 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:23:18.333 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:18.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:18.472 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:18.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:18.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:18.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:18.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:18 smithi038 ceph-mon[29779]: pgmap v968: 148 pgs: 148 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-04T16:23:18.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:18 smithi038 ceph-mon[38113]: pgmap v968: 148 pgs: 148 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-04T16:23:18.752 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:18.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:18.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:18 smithi176 ceph-mon[33619]: pgmap v968: 148 pgs: 148 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 822 op/s 2024-03-04T16:23:18.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:18.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:18.987 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:18.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 6 1 2024-03-04T16:23:19.012 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:19.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:19.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.155 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:19.256 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:19.301 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:19.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:19.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:19.607 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:19.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:19.742 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:19.844 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:19.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:19.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:19.979 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:20.015 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:20.117 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:20.143 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:20.245 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:20.270 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:20.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:20.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.421 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:20.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:20.522 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:20.548 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:20.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:20.649 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:20.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:20.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:20.827 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:20.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:20.964 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:20.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:21.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:21.098 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:21.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:20 smithi176 ceph-mon[33619]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-04T16:23:21.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:21.229 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:21.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:20 smithi038 ceph-mon[29779]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-04T16:23:21.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:20 smithi038 ceph-mon[38113]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-04T16:23:21.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:21.369 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:21.470 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:21.499 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:21.601 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:21.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:21.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:21.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:21.777 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:21.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:21.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:21.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:21.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:23:22.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:22.049 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:22.151 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:22.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:22.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:22.344 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:22.446 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:22.471 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:22.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:22.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:22.683 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:23:22.722 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:23:22.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:22.751 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:22.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:22.765 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:23:22.813 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:23:22.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:22.856 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:23:22.892 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:22.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:22.897 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:23:22.937 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:23:22.979 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:23:22.982 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:23:22.982 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:23:22.982 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:23:22.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:23:22.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:23:22.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:23:22.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:23:22.983 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:23:22.994 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:23.033 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:23.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:22 smithi176 ceph-mon[33619]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 960 op/s 2024-03-04T16:23:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:22 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4116462750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:23.135 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:23.186 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:23.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:22 smithi038 ceph-mon[29779]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 960 op/s 2024-03-04T16:23:23.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:22 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4116462750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:23.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:22 smithi038 ceph-mon[38113]: pgmap v970: 148 pgs: 148 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 960 op/s 2024-03-04T16:23:23.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:22 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4116462750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:23.288 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.357 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:23.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:23.485 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:23.587 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:23.725 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:23.736 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:23.764 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:23.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:23.866 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:23.903 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:23.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:24.005 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:24.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:24.147 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:24.202 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.203 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:24.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:24.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:24.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:24.465 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:24.504 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:24.606 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:24.635 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:24.672 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:24.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:24.782 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:24.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:24 smithi176 ceph-mon[33619]: pgmap v971: 148 pgs: 148 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:24.884 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:24.911 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:24.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[29779]: pgmap v971: 148 pgs: 148 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:24.961 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:24 smithi038 ceph-mon[38113]: pgmap v971: 148 pgs: 148 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:25.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:25.058 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:25.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:25.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:25.299 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:25.328 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:25.430 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:25.573 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.612 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:25.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.750 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:25.851 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:25.893 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:25.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:25.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:26.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:26.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.167 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:26.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:26.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:26.297 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:26.399 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:26.437 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:26.539 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:26.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:26.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:26.703 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:26.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:26.713 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:23:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:26 smithi038 ceph-mon[29779]: pgmap v972: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:26.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:26 smithi038 ceph-mon[38113]: pgmap v972: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:26.805 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.845 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:26.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:26.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:26 smithi176 ceph-mon[33619]: pgmap v972: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:26.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:26.986 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:26.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:27.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:27.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:27.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:27.272 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:27.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:27.409 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:27.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:27.542 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:27.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:27.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:27.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:27.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:27.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:27.912 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:27.949 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:27.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:28.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:28.073 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:28.175 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:28.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:28.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:28.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:28.339 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:28.441 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:28.477 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:28.578 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:28.603 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:28.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:28.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:28.740 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:28.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:28.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:28.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:28.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:28.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:28 smithi038 ceph-mon[29779]: pgmap v973: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:28.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:28 smithi038 ceph-mon[38113]: pgmap v973: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:29.016 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:29.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:28 smithi176 ceph-mon[33619]: pgmap v973: 148 pgs: 148 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 906 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:29.118 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:29.156 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:29.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:29.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.304 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:29.405 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:29.438 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:29.540 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:29.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:29.726 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:29.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:29.865 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:29.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:29.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:30.108 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.137 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:30.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:30.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:30.282 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:30.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:30.410 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:30.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:30.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:30.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:30.701 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:30.805 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:30.833 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[38113]: pgmap v974: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:23:30.834 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:30.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:30.843 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:23:30.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:30.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:30.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:31.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:31.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:31.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:31.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:30 smithi176 ceph-mon[33619]: pgmap v974: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:23:31.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1194509704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:30 smithi176 ceph-mon[33619]: from='client.35957 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.197 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:31.239 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:31.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[29779]: pgmap v974: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-03-04T16:23:31.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1194509704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[29779]: from='client.35957 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1194509704' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:30 smithi038 ceph-mon[38113]: from='client.35957 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:23:31.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:31.372 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:31.474 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.517 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:31.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:31.618 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.658 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:31.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:31.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:31.802 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:31.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:31.946 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:31.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[29779]: from='client.35957 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[29779]: osdmap e448: 8 total, 8 up, 7 in 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[38113]: from='client.35957 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[38113]: osdmap e448: 8 total, 8 up, 7 in 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:31.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:31 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:32.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.078 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:32.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:32.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:31 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:31 smithi176 ceph-mon[33619]: from='client.35957 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:23:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:31 smithi176 ceph-mon[33619]: osdmap e448: 8 total, 8 up, 7 in 2024-03-04T16:23:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:31 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:31 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:32.179 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:32.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:32.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:32.337 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:32.440 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:32.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:32.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.605 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:32.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:32.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:32.744 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:32.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:32.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:32.979 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:33.010 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:33.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:32 smithi176 ceph-mon[33619]: pgmap v976: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:33.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:32 smithi176 ceph-mon[33619]: osdmap e449: 8 total, 8 up, 7 in 2024-03-04T16:23:33.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:33.147 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:33.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:32 smithi038 ceph-mon[29779]: pgmap v976: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:33.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:32 smithi038 ceph-mon[29779]: osdmap e449: 8 total, 8 up, 7 in 2024-03-04T16:23:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:32 smithi038 ceph-mon[38113]: pgmap v976: 148 pgs: 148 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:32 smithi038 ceph-mon[38113]: osdmap e449: 8 total, 8 up, 7 in 2024-03-04T16:23:33.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:33.275 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:33.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:33.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:33.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.419 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:33.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:33.521 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.551 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:33.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:33.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:33.689 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:33.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:33.742 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:33.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:33.823 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:33.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:33.891 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:33.926 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:33.955 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:33.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:34.057 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:34.087 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:34.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.239 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:34.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:34.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.381 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:34.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:34.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:34.510 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:34.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:34.657 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:34.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:34.791 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:34.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:34.918 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:34.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:35.021 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:35.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:35.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:35.232 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:35.233 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:34 smithi038 ceph-mon[29779]: pgmap v978: 148 pgs: 148 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:35.233 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:35.233 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:34 smithi038 ceph-mon[38113]: pgmap v978: 148 pgs: 148 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:35.291 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:35.321 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:35.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:34 smithi176 ceph-mon[33619]: pgmap v978: 148 pgs: 148 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:35.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:35.453 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:35.555 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:35.582 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:35.684 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:35.715 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:35.817 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:35.846 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:35.892 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-03-04T16:23:35.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:35.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:35.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:35.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:35.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:35.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[29779]: osdmap e450: 8 total, 8 up, 7 in 2024-03-04T16:23:35.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2533726551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[29779]: from='client.25888 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[38113]: osdmap e450: 8 total, 8 up, 7 in 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2533726551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:35.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:35 smithi038 ceph-mon[38113]: from='client.25888 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:36.091 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.120 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:36.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:36.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:36.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:36.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:35 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:35 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:35 smithi176 ceph-mon[33619]: osdmap e450: 8 total, 8 up, 7 in 2024-03-04T16:23:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2533726551' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:35 smithi176 ceph-mon[33619]: from='client.25888 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:23:36.372 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:36.414 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:36.516 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.560 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:36.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:36.661 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:36.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.688 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:36.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:36.717 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T16:23:36.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:23:36.789 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:36.817 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:36.919 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:36.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:36.983 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:36.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:37.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:37.131 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:37.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:37.165 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[29779]: pgmap v980: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:23:37.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[29779]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:23:37.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[29779]: osdmap e451: 8 total, 8 up, 7 in 2024-03-04T16:23:37.166 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[38113]: pgmap v980: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:23:37.166 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[38113]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:23:37.166 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:36 smithi038 ceph-mon[38113]: osdmap e451: 8 total, 8 up, 7 in 2024-03-04T16:23:37.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.286 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:37.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:37.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:36 smithi176 ceph-mon[33619]: pgmap v980: 148 pgs: 148 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:23:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:36 smithi176 ceph-mon[33619]: from='client.25888 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:23:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:36 smithi176 ceph-mon[33619]: osdmap e451: 8 total, 8 up, 7 in 2024-03-04T16:23:37.387 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:37.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.433 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:37.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:37.534 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:37.564 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:37.667 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:37.696 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:37.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:37.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.845 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:37.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:37.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:37.997 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:37.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:37.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:38.100 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.142 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:38.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:38.186 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:23:38.226 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:23:38.243 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:38.267 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:23:38.270 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:38.312 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:23:38.357 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:23:38.372 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:38.400 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:38.401 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:23:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:38.442 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:23:38.486 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:23:38.489 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:23:38.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:23:38.503 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:38.521 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:23:38.521 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:23:38.528 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:38.630 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.677 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:38.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:38.779 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.809 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:38.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:38.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:38.939 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:38.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:39.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.082 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:39.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:39.183 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:39.211 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: pgmap v982: 148 pgs: 148 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-04T16:23:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: osdmap e452: 8 total, 8 up, 7 in 2024-03-04T16:23:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1098422451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: pgmap v982: 148 pgs: 148 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: osdmap e452: 8 total, 8 up, 7 in 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1098422451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:39.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:39.342 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:39.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:39.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: pgmap v982: 148 pgs: 148 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-04T16:23:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: osdmap e452: 8 total, 8 up, 7 in 2024-03-04T16:23:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1098422451' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:39.444 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:39.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:39.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:39.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:39.632 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:39.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:39.775 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:39.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:23:39.877 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:39.914 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:39.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:40.016 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.045 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:40.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:40.146 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:40.174 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:40.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:40.276 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:40.321 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:40.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:40.459 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:40.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:40.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:40.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:40.754 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:40.856 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:40.888 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:40.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:40.989 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.019 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:41.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:41.123 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.162 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:41.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:41.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:40 smithi038 ceph-mon[29779]: pgmap v984: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:23:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:40 smithi038 ceph-mon[38113]: pgmap v984: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:23:41.263 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:41.291 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:41.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:41.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:40 smithi176 ceph-mon[33619]: pgmap v984: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.28k op/s 2024-03-04T16:23:41.393 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:41.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:23:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.431 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:41.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:41.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:41.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.609 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:41.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:41.710 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:41.740 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:41.841 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:41.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:41.868 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:41.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:41.971 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:42.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:42.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:42.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:42.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:42.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:42.239 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.288 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:42.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:42.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:42.427 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:42.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:23:42.529 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:42.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:42.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.720 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:42.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:42.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:42.877 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:42.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:42.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:43.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:43.124 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:43.126 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.151 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:43.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:43.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:42 smithi038 ceph-mon[29779]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 814 op/s 2024-03-04T16:23:43.216 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:42 smithi038 ceph-mon[38113]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 814 op/s 2024-03-04T16:23:43.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:43.294 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:43.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:42 smithi176 ceph-mon[33619]: pgmap v985: 148 pgs: 148 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 743 B/s rd, 0 B/s wr, 814 op/s 2024-03-04T16:23:43.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:43.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.455 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:43.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:43.471 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:43.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:43.587 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:43.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:43.714 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:43.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:43.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:43.860 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:43.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:43.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.008 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:44.109 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:44.132 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:23:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:23:44.148 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:44.250 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:44.278 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:44.380 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.407 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:44.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:44.508 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:44.550 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:44.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:44.687 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:44.789 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:44.826 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:44.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:44.964 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:44.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:45.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:45.092 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:45.194 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.240 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:45.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[29779]: pgmap v986: 148 pgs: 148 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:45.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[29779]: from='client.35972 v1:172.21.15.38:0/1436353109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:23:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[38113]: pgmap v986: 148 pgs: 148 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:44 smithi038 ceph-mon[38113]: from='client.35972 v1:172.21.15.38:0/1436353109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:23:45.341 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:45.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:44 smithi176 ceph-mon[33619]: pgmap v986: 148 pgs: 148 active+clean; 4.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:44 smithi176 ceph-mon[33619]: from='client.35972 v1:172.21.15.38:0/1436353109' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:23:45.370 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:45.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:45.471 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:45.505 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:45.607 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:45.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:45.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:45.777 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:45.879 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:45.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:45.906 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:45.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:46.007 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.037 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:46.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:46.139 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.170 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:46.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:46.271 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:46.317 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:46.419 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:46.446 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:46.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:46.584 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:46.605 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:46.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:46.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:46 smithi038 ceph-mon[29779]: pgmap v987: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:46.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:46 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:23:46.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:46 smithi038 ceph-mon[38113]: pgmap v987: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:46.704 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:46 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:23:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.749 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:46.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:46.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:46.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:46 smithi176 ceph-mon[33619]: pgmap v987: 148 pgs: 148 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-04T16:23:46.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:46 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:23:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:46.896 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:46.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:46.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:47.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:47.111 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":986,"stamp":"2024-03-04T16:23:43.752008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4513732982,"num_objects":69064,"num_object_clones":0,"num_object_copies":207192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69064,"num_whiteouts":0,"num_read":3018,"num_read_kb":2864,"num_write":146595,"num_write_kb":6859166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8805,"num_bytes_recovered":548685156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77792,"ondisk_log_size":77792,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19506236,"kb_used_data":13424576,"kb_used_omap":10,"kb_used_meta":6081333,"kb_avail":636709828,"statfs":{"total":671965249536,"available":651990863872,"internally_reserved":0,"allocated":13746765824,"data_stored":13743871196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":6227285101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,7,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-599588864,"num_objects":-9149,"num_object_clones":0,"num_object_copies":-27447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9149,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9149,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002179"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":195,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937424+0000","last_change":"2024-03-04T16:21:44.712801+0000","last_active":"2024-03-04T16:23:37.937424+0000","last_peered":"2024-03-04T16:23:37.937424+0000","last_clean":"2024-03-04T16:23:37.937424+0000","last_became_active":"2024-03-04T16:20:07.476108+0000","last_became_peered":"2024-03-04T16:20:07.476108+0000","last_unstale":"2024-03-04T16:23:37.937424+0000","last_undegraded":"2024-03-04T16:23:37.937424+0000","last_fullsized":"2024-03-04T16:23:37.937424+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.712721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.712721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":835,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939518+0000","last_change":"2024-03-04T16:21:45.018101+0000","last_active":"2024-03-04T16:23:37.939518+0000","last_peered":"2024-03-04T16:23:37.939518+0000","last_clean":"2024-03-04T16:23:37.939518+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:23:37.939518+0000","last_undegraded":"2024-03-04T16:23:37.939518+0000","last_fullsized":"2024-03-04T16:23:37.939518+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:45.018020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:21:45.018020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.386869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":706,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.944236+0000","last_change":"2024-03-04T16:22:06.502649+0000","last_active":"2024-03-04T16:23:37.944236+0000","last_peered":"2024-03-04T16:23:37.944236+0000","last_clean":"2024-03-04T16:23:37.944236+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:23:37.944236+0000","last_undegraded":"2024-03-04T16:23:37.944236+0000","last_fullsized":"2024-03-04T16:23:37.944236+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.502545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.502545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":116,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.931570+0000","last_change":"2024-03-04T16:21:45.470673+0000","last_active":"2024-03-04T16:23:37.931570+0000","last_peered":"2024-03-04T16:23:37.931570+0000","last_clean":"2024-03-04T16:23:37.931570+0000","last_became_active":"2024-03-04T16:21:45.423494+0000","last_became_peered":"2024-03-04T16:21:45.423494+0000","last_unstale":"2024-03-04T16:23:37.931570+0000","last_undegraded":"2024-03-04T16:23:37.931570+0000","last_fullsized":"2024-03-04T16:23:37.931570+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:21:38.860021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.860021+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":835,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937978+0000","last_change":"2024-03-04T16:21:45.429466+0000","last_active":"2024-03-04T16:23:37.937978+0000","last_peered":"2024-03-04T16:23:37.937978+0000","last_clean":"2024-03-04T16:23:37.937978+0000","last_became_active":"2024-03-04T16:21:45.427819+0000","last_became_peered":"2024-03-04T16:21:45.427819+0000","last_unstale":"2024-03-04T16:23:37.937978+0000","last_undegraded":"2024-03-04T16:23:37.937978+0000","last_fullsized":"2024-03-04T16:23:37.937978+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.058061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.058061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074127+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.931659+0000","last_change":"2024-03-04T16:22:56.684863+0000","last_active":"2024-03-04T16:23:37.931659+0000","last_peered":"2024-03-04T16:23:37.931659+0000","last_clean":"2024-03-04T16:23:37.931659+0000","last_became_active":"2024-03-04T16:21:45.422482+0000","last_became_peered":"2024-03-04T16:21:45.422482+0000","last_unstale":"2024-03-04T16:23:37.931659+0000","last_undegraded":"2024-03-04T16:23:37.931659+0000","last_fullsized":"2024-03-04T16:23:37.931659+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.684724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.684724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.493825+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938364+0000","last_change":"2024-03-04T16:23:08.925901+0000","last_active":"2024-03-04T16:23:37.938364+0000","last_peered":"2024-03-04T16:23:37.938364+0000","last_clean":"2024-03-04T16:23:37.938364+0000","last_became_active":"2024-03-04T16:21:45.422756+0000","last_became_peered":"2024-03-04T16:21:45.422756+0000","last_unstale":"2024-03-04T16:23:37.938364+0000","last_undegraded":"2024-03-04T16:23:37.938364+0000","last_fullsized":"2024-03-04T16:23:37.938364+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.925783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.299467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'39","reported_seq":837,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.928721+0000","last_change":"2024-03-04T16:22:47.640816+0000","last_active":"2024-03-04T16:23:37.928721+0000","last_peered":"2024-03-04T16:23:37.928721+0000","last_clean":"2024-03-04T16:23:37.928721+0000","last_became_active":"2024-03-04T16:21:25.444226+0000","last_became_peered":"2024-03-04T16:21:25.444226+0000","last_unstale":"2024-03-04T16:23:37.928721+0000","last_undegraded":"2024-03-04T16:23:37.928721+0000","last_fullsized":"2024-03-04T16:23:37.928721+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:22:47.640638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.640638+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.818716+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":695,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932305+0000","last_change":"2024-03-04T16:22:50.786398+0000","last_active":"2024-03-04T16:23:37.932305+0000","last_peered":"2024-03-04T16:23:37.932305+0000","last_clean":"2024-03-04T16:23:37.932305+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:23:37.932305+0000","last_undegraded":"2024-03-04T16:23:37.932305+0000","last_fullsized":"2024-03-04T16:23:37.932305+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.786274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.786274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.345657+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":834,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936439+0000","last_change":"2024-03-04T16:22:06.888682+0000","last_active":"2024-03-04T16:23:37.936439+0000","last_peered":"2024-03-04T16:23:37.936439+0000","last_clean":"2024-03-04T16:23:37.936439+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:23:37.936439+0000","last_undegraded":"2024-03-04T16:23:37.936439+0000","last_fullsized":"2024-03-04T16:23:37.936439+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.888592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093211+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":868,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936776+0000","last_change":"2024-03-04T16:21:45.443994+0000","last_active":"2024-03-04T16:23:37.936776+0000","last_peered":"2024-03-04T16:23:37.936776+0000","last_clean":"2024-03-04T16:23:37.936776+0000","last_became_active":"2024-03-04T16:21:45.443239+0000","last_became_peered":"2024-03-04T16:21:45.443239+0000","last_unstale":"2024-03-04T16:23:37.936776+0000","last_undegraded":"2024-03-04T16:23:37.936776+0000","last_fullsized":"2024-03-04T16:23:37.936776+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:36.877837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:36.877837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093178+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'25","reported_seq":660,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.946534+0000","last_change":"2024-03-04T16:22:09.726711+0000","last_active":"2024-03-04T16:23:37.946534+0000","last_peered":"2024-03-04T16:23:37.946534+0000","last_clean":"2024-03-04T16:23:37.946534+0000","last_became_active":"2024-03-04T16:20:07.491874+0000","last_became_peered":"2024-03-04T16:20:07.491874+0000","last_unstale":"2024-03-04T16:23:37.946534+0000","last_undegraded":"2024-03-04T16:23:37.946534+0000","last_fullsized":"2024-03-04T16:23:37.946534+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:09.726573+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.726573+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.289179+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":838,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936103+0000","last_change":"2024-03-04T16:20:56.455146+0000","last_active":"2024-03-04T16:23:37.936103+0000","last_peered":"2024-03-04T16:23:37.936103+0000","last_clean":"2024-03-04T16:23:37.936103+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:23:37.936103+0000","last_undegraded":"2024-03-04T16:23:37.936103+0000","last_fullsized":"2024-03-04T16:23:37.936103+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:56.455020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:20:56.455020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093059+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":857,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929981+0000","last_change":"2024-03-04T16:22:46.297643+0000","last_active":"2024-03-04T16:23:37.929981+0000","last_peered":"2024-03-04T16:23:37.929981+0000","last_clean":"2024-03-04T16:23:37.929981+0000","last_became_active":"2024-03-04T16:20:07.494164+0000","last_became_peered":"2024-03-04T16:20:07.494164+0000","last_unstale":"2024-03-04T16:23:37.929981+0000","last_undegraded":"2024-03-04T16:23:37.929981+0000","last_fullsized":"2024-03-04T16:23:37.929981+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.297535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.297535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.975907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":860,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936914+0000","last_change":"2024-03-04T16:21:45.422865+0000","last_active":"2024-03-04T16:23:37.936914+0000","last_peered":"2024-03-04T16:23:37.936914+0000","last_clean":"2024-03-04T16:23:37.936914+0000","last_became_active":"2024-03-04T16:21:45.422460+0000","last_became_peered":"2024-03-04T16:21:45.422460+0000","last_unstale":"2024-03-04T16:23:37.936914+0000","last_undegraded":"2024-03-04T16:23:37.936914+0000","last_fullsized":"2024-03-04T16:23:37.936914+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:55.405634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:20:55.405634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"441'11","reported_seq":628,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937859+0000","last_change":"2024-03-04T16:23:00.985610+0000","last_active":"2024-03-04T16:23:37.937859+0000","last_peered":"2024-03-04T16:23:37.937859+0000","last_clean":"2024-03-04T16:23:37.937859+0000","last_became_active":"2024-03-04T16:20:07.482406+0000","last_became_peered":"2024-03-04T16:20:07.482406+0000","last_unstale":"2024-03-04T16:23:37.937859+0000","last_undegraded":"2024-03-04T16:23:37.937859+0000","last_fullsized":"2024-03-04T16:23:37.937859+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'11","last_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_deep_scrub":"441'11","last_deep_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.985229+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.484493+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932556+0000","last_change":"2024-03-04T16:22:59.694873+0000","last_active":"2024-03-04T16:23:37.932556+0000","last_peered":"2024-03-04T16:23:37.932556+0000","last_clean":"2024-03-04T16:23:37.932556+0000","last_became_active":"2024-03-04T16:21:45.425723+0000","last_became_peered":"2024-03-04T16:21:45.425723+0000","last_unstale":"2024-03-04T16:23:37.932556+0000","last_undegraded":"2024-03-04T16:23:37.932556+0000","last_fullsized":"2024-03-04T16:23:37.932556+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:59.694757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:59.694757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:19.218070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":849,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940289+0000","last_change":"2024-03-04T16:21:41.913454+0000","last_active":"2024-03-04T16:23:37.940289+0000","last_peered":"2024-03-04T16:23:37.940289+0000","last_clean":"2024-03-04T16:23:37.940289+0000","last_became_active":"2024-03-04T16:20:07.497650+0000","last_became_peered":"2024-03-04T16:20:07.497650+0000","last_unstale":"2024-03-04T16:23:37.940289+0000","last_undegraded":"2024-03-04T16:23:37.940289+0000","last_fullsized":"2024-03-04T16:23:37.940289+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:41.913323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.913323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.386774+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":693,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932861+0000","last_change":"2024-03-04T16:22:45.654818+0000","last_active":"2024-03-04T16:23:37.932861+0000","last_peered":"2024-03-04T16:23:37.932861+0000","last_clean":"2024-03-04T16:23:37.932861+0000","last_became_active":"2024-03-04T16:20:07.498539+0000","last_became_peered":"2024-03-04T16:20:07.498539+0000","last_unstale":"2024-03-04T16:23:37.932861+0000","last_undegraded":"2024-03-04T16:23:37.932861+0000","last_fullsized":"2024-03-04T16:23:37.932861+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.654354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.654354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.175429+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":282,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930041+0000","last_change":"2024-03-04T16:21:55.824199+0000","last_active":"2024-03-04T16:23:37.930041+0000","last_peered":"2024-03-04T16:23:37.930041+0000","last_clean":"2024-03-04T16:23:37.930041+0000","last_became_active":"2024-03-04T16:19:04.771891+0000","last_became_peered":"2024-03-04T16:19:04.771891+0000","last_unstale":"2024-03-04T16:23:37.930041+0000","last_undegraded":"2024-03-04T16:23:37.930041+0000","last_fullsized":"2024-03-04T16:23:37.930041+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:21:55.824051+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:21:55.824051+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.215243+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932658+0000","last_change":"2024-03-04T16:22:43.806087+0000","last_active":"2024-03-04T16:23:37.932658+0000","last_peered":"2024-03-04T16:23:37.932658+0000","last_clean":"2024-03-04T16:23:37.932658+0000","last_became_active":"2024-03-04T16:21:45.420331+0000","last_became_peered":"2024-03-04T16:21:45.420331+0000","last_unstale":"2024-03-04T16:23:37.932658+0000","last_undegraded":"2024-03-04T16:23:37.932658+0000","last_fullsized":"2024-03-04T16:23:37.932658+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.805945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.805945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.468231+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":795,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939102+0000","last_change":"2024-03-04T16:21:42.722963+0000","last_active":"2024-03-04T16:23:37.939102+0000","last_peered":"2024-03-04T16:23:37.939102+0000","last_clean":"2024-03-04T16:23:37.939102+0000","last_became_active":"2024-03-04T16:20:07.474386+0000","last_became_peered":"2024-03-04T16:20:07.474386+0000","last_unstale":"2024-03-04T16:23:37.939102+0000","last_undegraded":"2024-03-04T16:23:37.939102+0000","last_fullsized":"2024-03-04T16:23:37.939102+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:42.722804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:42.722804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074180+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":811,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939919+0000","last_change":"2024-03-04T16:22:56.344093+0000","last_active":"2024-03-04T16:23:37.939919+0000","last_peered":"2024-03-04T16:23:37.939919+0000","last_clean":"2024-03-04T16:23:37.939919+0000","last_became_active":"2024-03-04T16:20:07.503831+0000","last_became_peered":"2024-03-04T16:20:07.503831+0000","last_unstale":"2024-03-04T16:23:37.939919+0000","last_undegraded":"2024-03-04T16:23:37.939919+0000","last_fullsized":"2024-03-04T16:23:37.939919+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.343792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.343792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.303419+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":556,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939988+0000","last_change":"2024-03-04T16:21:45.475840+0000","last_active":"2024-03-04T16:23:37.939988+0000","last_peered":"2024-03-04T16:23:37.939988+0000","last_clean":"2024-03-04T16:23:37.939988+0000","last_became_active":"2024-03-04T16:21:45.441225+0000","last_became_peered":"2024-03-04T16:21:45.441225+0000","last_unstale":"2024-03-04T16:23:37.939988+0000","last_undegraded":"2024-03-04T16:23:37.939988+0000","last_fullsized":"2024-03-04T16:23:37.939988+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:21:40.886581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.886581+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.386841+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930375+0000","last_change":"2024-03-04T16:22:45.271732+0000","last_active":"2024-03-04T16:23:37.930375+0000","last_peered":"2024-03-04T16:23:37.930375+0000","last_clean":"2024-03-04T16:23:37.930375+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:23:37.930375+0000","last_undegraded":"2024-03-04T16:23:37.930375+0000","last_fullsized":"2024-03-04T16:23:37.930375+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:22:45.271595+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.271595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:56.454962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"412'13","reported_seq":124,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932531+0000","last_change":"2024-03-04T16:21:45.443536+0000","last_active":"2024-03-04T16:23:37.932531+0000","last_peered":"2024-03-04T16:23:37.932531+0000","last_clean":"2024-03-04T16:23:37.932531+0000","last_became_active":"2024-03-04T16:21:45.416857+0000","last_became_peered":"2024-03-04T16:21:45.416857+0000","last_unstale":"2024-03-04T16:23:37.932531+0000","last_undegraded":"2024-03-04T16:23:37.932531+0000","last_fullsized":"2024-03-04T16:23:37.932531+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'12","last_scrub_stamp":"2024-03-04T16:21:38.843730+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.843730+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":872,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939779+0000","last_change":"2024-03-04T16:22:43.057543+0000","last_active":"2024-03-04T16:23:37.939779+0000","last_peered":"2024-03-04T16:23:37.939779+0000","last_clean":"2024-03-04T16:23:37.939779+0000","last_became_active":"2024-03-04T16:21:45.437760+0000","last_became_peered":"2024-03-04T16:21:45.437760+0000","last_unstale":"2024-03-04T16:23:37.939779+0000","last_undegraded":"2024-03-04T16:23:37.939779+0000","last_fullsized":"2024-03-04T16:23:37.939779+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:43.057419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.057419+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:01.275240+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":739,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.934555+0000","last_change":"2024-03-04T16:23:02.578786+0000","last_active":"2024-03-04T16:23:37.934555+0000","last_peered":"2024-03-04T16:23:37.934555+0000","last_clean":"2024-03-04T16:23:37.934555+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:23:37.934555+0000","last_undegraded":"2024-03-04T16:23:37.934555+0000","last_fullsized":"2024-03-04T16:23:37.934555+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:23:02.578648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.578648+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.686651+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":845,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940380+0000","last_change":"2024-03-04T16:23:06.666597+0000","last_active":"2024-03-04T16:23:37.940380+0000","last_peered":"2024-03-04T16:23:37.940380+0000","last_clean":"2024-03-04T16:23:37.940380+0000","last_became_active":"2024-03-04T16:20:07.477199+0000","last_became_peered":"2024-03-04T16:20:07.477199+0000","last_unstale":"2024-03-04T16:23:37.940380+0000","last_undegraded":"2024-03-04T16:23:37.940380+0000","last_fullsized":"2024-03-04T16:23:37.940380+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:06.666475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:08.786009+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.666475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:33.618544+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":372,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.934074+0000","last_change":"2024-03-04T16:22:57.654317+0000","last_active":"2024-03-04T16:23:37.934074+0000","last_peered":"2024-03-04T16:23:37.934074+0000","last_clean":"2024-03-04T16:23:37.934074+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:23:37.934074+0000","last_undegraded":"2024-03-04T16:23:37.934074+0000","last_fullsized":"2024-03-04T16:23:37.934074+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:57.653816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.653816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.989031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":847,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937832+0000","last_change":"2024-03-04T16:21:38.659764+0000","last_active":"2024-03-04T16:23:37.937832+0000","last_peered":"2024-03-04T16:23:37.937832+0000","last_clean":"2024-03-04T16:23:37.937832+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:23:37.937832+0000","last_undegraded":"2024-03-04T16:23:37.937832+0000","last_fullsized":"2024-03-04T16:23:37.937832+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:38.659652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.659652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074154+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'27","reported_seq":129,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932605+0000","last_change":"2024-03-04T16:22:42.533910+0000","last_active":"2024-03-04T16:23:37.932605+0000","last_peered":"2024-03-04T16:23:37.932605+0000","last_clean":"2024-03-04T16:23:37.932605+0000","last_became_active":"2024-03-04T16:21:45.434840+0000","last_became_peered":"2024-03-04T16:21:45.434840+0000","last_unstale":"2024-03-04T16:23:37.932605+0000","last_undegraded":"2024-03-04T16:23:37.932605+0000","last_fullsized":"2024-03-04T16:23:37.932605+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:42.533824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.533824+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.764785+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":865,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930120+0000","last_change":"2024-03-04T16:22:09.161333+0000","last_active":"2024-03-04T16:23:37.930120+0000","last_peered":"2024-03-04T16:23:37.930120+0000","last_clean":"2024-03-04T16:23:37.930120+0000","last_became_active":"2024-03-04T16:20:07.477661+0000","last_became_peered":"2024-03-04T16:20:07.477661+0000","last_unstale":"2024-03-04T16:23:37.930120+0000","last_undegraded":"2024-03-04T16:23:37.930120+0000","last_fullsized":"2024-03-04T16:23:37.930120+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.161217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.161217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279483+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932214+0000","last_change":"2024-03-04T16:23:08.842921+0000","last_active":"2024-03-04T16:23:37.932214+0000","last_peered":"2024-03-04T16:23:37.932214+0000","last_clean":"2024-03-04T16:23:37.932214+0000","last_became_active":"2024-03-04T16:21:45.414027+0000","last_became_peered":"2024-03-04T16:21:45.414027+0000","last_unstale":"2024-03-04T16:23:37.932214+0000","last_undegraded":"2024-03-04T16:23:37.932214+0000","last_fullsized":"2024-03-04T16:23:37.932214+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.842842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.842842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:21.737638+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":687,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.944134+0000","last_change":"2024-03-04T16:22:10.615994+0000","last_active":"2024-03-04T16:23:37.944134+0000","last_peered":"2024-03-04T16:23:37.944134+0000","last_clean":"2024-03-04T16:23:37.944134+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:23:37.944134+0000","last_undegraded":"2024-03-04T16:23:37.944134+0000","last_fullsized":"2024-03-04T16:23:37.944134+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:10.615811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:10.615811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656892+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"441'12","reported_seq":153,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.928241+0000","last_change":"2024-03-04T16:22:50.599109+0000","last_active":"2024-03-04T16:23:37.928241+0000","last_peered":"2024-03-04T16:23:37.928241+0000","last_clean":"2024-03-04T16:23:37.928241+0000","last_became_active":"2024-03-04T16:21:45.423994+0000","last_became_peered":"2024-03-04T16:21:45.423994+0000","last_unstale":"2024-03-04T16:23:37.928241+0000","last_undegraded":"2024-03-04T16:23:37.928241+0000","last_fullsized":"2024-03-04T16:23:37.928241+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:22:50.598967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.598967+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.026066+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":853,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938077+0000","last_change":"2024-03-04T16:23:03.173401+0000","last_active":"2024-03-04T16:23:37.938077+0000","last_peered":"2024-03-04T16:23:37.938077+0000","last_clean":"2024-03-04T16:23:37.938077+0000","last_became_active":"2024-03-04T16:21:45.429844+0000","last_became_peered":"2024-03-04T16:21:45.429844+0000","last_unstale":"2024-03-04T16:23:37.938077+0000","last_undegraded":"2024-03-04T16:23:37.938077+0000","last_fullsized":"2024-03-04T16:23:37.938077+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.173174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.173174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.851084+0000","scrub_duration":244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930068+0000","last_change":"2024-03-04T16:22:54.333690+0000","last_active":"2024-03-04T16:23:37.930068+0000","last_peered":"2024-03-04T16:23:37.930068+0000","last_clean":"2024-03-04T16:23:37.930068+0000","last_became_active":"2024-03-04T16:21:45.415087+0000","last_became_peered":"2024-03-04T16:21:45.415087+0000","last_unstale":"2024-03-04T16:23:37.930068+0000","last_undegraded":"2024-03-04T16:23:37.930068+0000","last_fullsized":"2024-03-04T16:23:37.930068+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.333535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.333535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.663716+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":850,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937400+0000","last_change":"2024-03-04T16:22:41.028799+0000","last_active":"2024-03-04T16:23:37.937400+0000","last_peered":"2024-03-04T16:23:37.937400+0000","last_clean":"2024-03-04T16:23:37.937400+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:23:37.937400+0000","last_undegraded":"2024-03-04T16:23:37.937400+0000","last_fullsized":"2024-03-04T16:23:37.937400+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.028680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.028680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.495192+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'27","reported_seq":910,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938267+0000","last_change":"2024-03-04T16:22:55.927787+0000","last_active":"2024-03-04T16:23:37.938267+0000","last_peered":"2024-03-04T16:23:37.938267+0000","last_clean":"2024-03-04T16:23:37.938267+0000","last_became_active":"2024-03-04T16:21:45.428057+0000","last_became_peered":"2024-03-04T16:21:45.428057+0000","last_unstale":"2024-03-04T16:23:37.938267+0000","last_undegraded":"2024-03-04T16:23:37.938267+0000","last_fullsized":"2024-03-04T16:23:37.938267+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:22:55.927235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:55.927235+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.688732+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":687,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932376+0000","last_change":"2024-03-04T16:22:46.734503+0000","last_active":"2024-03-04T16:23:37.932376+0000","last_peered":"2024-03-04T16:23:37.932376+0000","last_clean":"2024-03-04T16:23:37.932376+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:23:37.932376+0000","last_undegraded":"2024-03-04T16:23:37.932376+0000","last_fullsized":"2024-03-04T16:23:37.932376+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.734392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.734392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:10.402153+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":476,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936363+0000","last_change":"2024-03-04T16:21:25.449367+0000","last_active":"2024-03-04T16:23:37.936363+0000","last_peered":"2024-03-04T16:23:37.936363+0000","last_clean":"2024-03-04T16:23:37.936363+0000","last_became_active":"2024-03-04T16:21:25.449034+0000","last_became_peered":"2024-03-04T16:21:25.449034+0000","last_unstale":"2024-03-04T16:23:37.936363+0000","last_undegraded":"2024-03-04T16:23:37.936363+0000","last_fullsized":"2024-03-04T16:23:37.936363+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:52.030278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:20:52.030278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":857,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930765+0000","last_change":"2024-03-04T16:21:58.944553+0000","last_active":"2024-03-04T16:23:37.930765+0000","last_peered":"2024-03-04T16:23:37.930765+0000","last_clean":"2024-03-04T16:23:37.930765+0000","last_became_active":"2024-03-04T16:20:07.498410+0000","last_became_peered":"2024-03-04T16:20:07.498410+0000","last_unstale":"2024-03-04T16:23:37.930765+0000","last_undegraded":"2024-03-04T16:23:37.930765+0000","last_fullsized":"2024-03-04T16:23:37.930765+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:58.944477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:58.944477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'25","reported_seq":76,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937318+0000","last_change":"2024-03-04T16:22:42.023805+0000","last_active":"2024-03-04T16:23:37.937318+0000","last_peered":"2024-03-04T16:23:37.937318+0000","last_clean":"2024-03-04T16:23:37.937318+0000","last_became_active":"2024-03-04T16:22:24.825469+0000","last_became_peered":"2024-03-04T16:22:24.825469+0000","last_unstale":"2024-03-04T16:23:37.937318+0000","last_undegraded":"2024-03-04T16:23:37.937318+0000","last_fullsized":"2024-03-04T16:23:37.937318+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:42.023633+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.023633+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:48.377369+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":278,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.933334+0000","last_change":"2024-03-04T16:22:44.719886+0000","last_active":"2024-03-04T16:23:37.933334+0000","last_peered":"2024-03-04T16:23:37.933334+0000","last_clean":"2024-03-04T16:23:37.933334+0000","last_became_active":"2024-03-04T16:21:25.446654+0000","last_became_peered":"2024-03-04T16:21:25.446654+0000","last_unstale":"2024-03-04T16:23:37.933334+0000","last_undegraded":"2024-03-04T16:23:37.933334+0000","last_fullsized":"2024-03-04T16:23:37.933334+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.719762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.719762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.792226+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":843,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940068+0000","last_change":"2024-03-04T16:22:47.060545+0000","last_active":"2024-03-04T16:23:37.940068+0000","last_peered":"2024-03-04T16:23:37.940068+0000","last_clean":"2024-03-04T16:23:37.940068+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:23:37.940068+0000","last_undegraded":"2024-03-04T16:23:37.940068+0000","last_fullsized":"2024-03-04T16:23:37.940068+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.060423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.060423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:56.705586+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":809,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930389+0000","last_change":"2024-03-04T16:21:45.416172+0000","last_active":"2024-03-04T16:23:37.930389+0000","last_peered":"2024-03-04T16:23:37.930389+0000","last_clean":"2024-03-04T16:23:37.930389+0000","last_became_active":"2024-03-04T16:21:45.415525+0000","last_became_peered":"2024-03-04T16:21:45.415525+0000","last_unstale":"2024-03-04T16:23:37.930389+0000","last_undegraded":"2024-03-04T16:23:37.930389+0000","last_fullsized":"2024-03-04T16:23:37.930389+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:37.800150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:37.800150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.277632+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":129,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932206+0000","last_change":"2024-03-04T16:22:52.714265+0000","last_active":"2024-03-04T16:23:37.932206+0000","last_peered":"2024-03-04T16:23:37.932206+0000","last_clean":"2024-03-04T16:23:37.932206+0000","last_became_active":"2024-03-04T16:21:45.437462+0000","last_became_peered":"2024-03-04T16:21:45.437462+0000","last_unstale":"2024-03-04T16:23:37.932206+0000","last_undegraded":"2024-03-04T16:23:37.932206+0000","last_fullsized":"2024-03-04T16:23:37.932206+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:22:52.714182+0000","last_deep_scrub":"316'24","last_deep_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.714182+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.278985+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":720,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930140+0000","last_change":"2024-03-04T16:23:11.346983+0000","last_active":"2024-03-04T16:23:37.930140+0000","last_peered":"2024-03-04T16:23:37.930140+0000","last_clean":"2024-03-04T16:23:37.930140+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:23:37.930140+0000","last_undegraded":"2024-03-04T16:23:37.930140+0000","last_fullsized":"2024-03-04T16:23:37.930140+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:11.346910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:11.346910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.225627+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":1116,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936897+0000","last_change":"2024-03-04T16:22:04.841240+0000","last_active":"2024-03-04T16:23:37.936897+0000","last_peered":"2024-03-04T16:23:37.936897+0000","last_clean":"2024-03-04T16:23:37.936897+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:23:37.936897+0000","last_undegraded":"2024-03-04T16:23:37.936897+0000","last_fullsized":"2024-03-04T16:23:37.936897+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:22:04.841055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.841055+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093196+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":915,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930763+0000","last_change":"2024-03-04T16:23:03.314418+0000","last_active":"2024-03-04T16:23:37.930763+0000","last_peered":"2024-03-04T16:23:37.930763+0000","last_clean":"2024-03-04T16:23:37.930763+0000","last_became_active":"2024-03-04T16:21:45.413361+0000","last_became_peered":"2024-03-04T16:21:45.413361+0000","last_unstale":"2024-03-04T16:23:37.930763+0000","last_undegraded":"2024-03-04T16:23:37.930763+0000","last_fullsized":"2024-03-04T16:23:37.930763+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:23:03.314294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.314294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:26.292076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'30","reported_seq":570,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940741+0000","last_change":"2024-03-04T16:21:35.752373+0000","last_active":"2024-03-04T16:23:37.940741+0000","last_peered":"2024-03-04T16:23:37.940741+0000","last_clean":"2024-03-04T16:23:37.940741+0000","last_became_active":"2024-03-04T16:21:25.452317+0000","last_became_peered":"2024-03-04T16:21:25.452317+0000","last_unstale":"2024-03-04T16:23:37.940741+0000","last_undegraded":"2024-03-04T16:23:37.940741+0000","last_fullsized":"2024-03-04T16:23:37.940741+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:21:35.752284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:35.752284+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.386807+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":799,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930731+0000","last_change":"2024-03-04T16:23:09.663195+0000","last_active":"2024-03-04T16:23:37.930731+0000","last_peered":"2024-03-04T16:23:37.930731+0000","last_clean":"2024-03-04T16:23:37.930731+0000","last_became_active":"2024-03-04T16:20:07.486320+0000","last_became_peered":"2024-03-04T16:20:07.486320+0000","last_unstale":"2024-03-04T16:23:37.930731+0000","last_undegraded":"2024-03-04T16:23:37.930731+0000","last_fullsized":"2024-03-04T16:23:37.930731+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:09.662860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.662860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.759094+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"338'11","reported_seq":1108,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939297+0000","last_change":"2024-03-04T16:22:57.950248+0000","last_active":"2024-03-04T16:23:37.939297+0000","last_peered":"2024-03-04T16:23:37.939297+0000","last_clean":"2024-03-04T16:23:37.939297+0000","last_became_active":"2024-03-04T16:19:04.780989+0000","last_became_peered":"2024-03-04T16:19:04.780989+0000","last_unstale":"2024-03-04T16:23:37.939297+0000","last_undegraded":"2024-03-04T16:23:37.939297+0000","last_fullsized":"2024-03-04T16:23:37.939297+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:57.950104+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.950104+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:06.619332+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":745,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929803+0000","last_change":"2024-03-04T16:22:00.982774+0000","last_active":"2024-03-04T16:23:37.929803+0000","last_peered":"2024-03-04T16:23:37.929803+0000","last_clean":"2024-03-04T16:23:37.929803+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:23:37.929803+0000","last_undegraded":"2024-03-04T16:23:37.929803+0000","last_fullsized":"2024-03-04T16:23:37.929803+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:00.982672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:00.982672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"441'21","reported_seq":880,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937644+0000","last_change":"2024-03-04T16:22:56.048086+0000","last_active":"2024-03-04T16:23:37.937644+0000","last_peered":"2024-03-04T16:23:37.937644+0000","last_clean":"2024-03-04T16:23:37.937644+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:23:37.937644+0000","last_undegraded":"2024-03-04T16:23:37.937644+0000","last_fullsized":"2024-03-04T16:23:37.937644+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:22:56.047875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.047875+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.270041+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":254,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930201+0000","last_change":"2024-03-04T16:21:54.866369+0000","last_active":"2024-03-04T16:23:37.930201+0000","last_peered":"2024-03-04T16:23:37.930201+0000","last_clean":"2024-03-04T16:23:37.930201+0000","last_became_active":"2024-03-04T16:20:07.491512+0000","last_became_peered":"2024-03-04T16:20:07.491512+0000","last_unstale":"2024-03-04T16:23:37.930201+0000","last_undegraded":"2024-03-04T16:23:37.930201+0000","last_fullsized":"2024-03-04T16:23:37.930201+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:54.866103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:54.866103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.315229+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":827,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939398+0000","last_change":"2024-03-04T16:22:41.839730+0000","last_active":"2024-03-04T16:23:37.939398+0000","last_peered":"2024-03-04T16:23:37.939398+0000","last_clean":"2024-03-04T16:23:37.939398+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:23:37.939398+0000","last_undegraded":"2024-03-04T16:23:37.939398+0000","last_fullsized":"2024-03-04T16:23:37.939398+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.839621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.839621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.180908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929930+0000","last_change":"2024-03-04T16:21:45.415202+0000","last_active":"2024-03-04T16:23:37.929930+0000","last_peered":"2024-03-04T16:23:37.929930+0000","last_clean":"2024-03-04T16:23:37.929930+0000","last_became_active":"2024-03-04T16:21:45.414717+0000","last_became_peered":"2024-03-04T16:21:45.414717+0000","last_unstale":"2024-03-04T16:23:37.929930+0000","last_undegraded":"2024-03-04T16:23:37.929930+0000","last_fullsized":"2024-03-04T16:23:37.929930+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_clean_scrub_stamp":"2024-03-04T16:19:30.702539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.277536+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'21","reported_seq":715,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.934615+0000","last_change":"2024-03-04T16:21:40.428230+0000","last_active":"2024-03-04T16:23:37.934615+0000","last_peered":"2024-03-04T16:23:37.934615+0000","last_clean":"2024-03-04T16:23:37.934615+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:23:37.934615+0000","last_undegraded":"2024-03-04T16:23:37.934615+0000","last_fullsized":"2024-03-04T16:23:37.934615+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:21:40.428083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.428083+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":857,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930342+0000","last_change":"2024-03-04T16:21:45.412628+0000","last_active":"2024-03-04T16:23:37.930342+0000","last_peered":"2024-03-04T16:23:37.930342+0000","last_clean":"2024-03-04T16:23:37.930342+0000","last_became_active":"2024-03-04T16:21:45.411986+0000","last_became_peered":"2024-03-04T16:21:45.411986+0000","last_unstale":"2024-03-04T16:23:37.930342+0000","last_undegraded":"2024-03-04T16:23:37.930342+0000","last_fullsized":"2024-03-04T16:23:37.930342+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:59.610360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:20:59.610360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.277568+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":834,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937321+0000","last_change":"2024-03-04T16:20:59.503150+0000","last_active":"2024-03-04T16:23:37.937321+0000","last_peered":"2024-03-04T16:23:37.937321+0000","last_clean":"2024-03-04T16:23:37.937321+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:23:37.937321+0000","last_undegraded":"2024-03-04T16:23:37.937321+0000","last_fullsized":"2024-03-04T16:23:37.937321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:59.503052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:20:59.503052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093123+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":97,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.931771+0000","last_change":"2024-03-04T16:21:45.414340+0000","last_active":"2024-03-04T16:23:37.931771+0000","last_peered":"2024-03-04T16:23:37.931771+0000","last_clean":"2024-03-04T16:23:37.931771+0000","last_became_active":"2024-03-04T16:21:45.413989+0000","last_became_peered":"2024-03-04T16:21:45.413989+0000","last_unstale":"2024-03-04T16:23:37.931771+0000","last_undegraded":"2024-03-04T16:23:37.931771+0000","last_fullsized":"2024-03-04T16:23:37.931771+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:39.439330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:21:39.439330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'21","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939043+0000","last_change":"2024-03-04T16:22:44.928303+0000","last_active":"2024-03-04T16:23:37.939043+0000","last_peered":"2024-03-04T16:23:37.939043+0000","last_clean":"2024-03-04T16:23:37.939043+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:23:37.939043+0000","last_undegraded":"2024-03-04T16:23:37.939043+0000","last_fullsized":"2024-03-04T16:23:37.939043+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:22:44.928118+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.928118+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:46.528175+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"452'9175","reported_seq":9288,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:43.104991+0000","last_change":"2024-03-04T16:21:25.449309+0000","last_active":"2024-03-04T16:23:43.104991+0000","last_peered":"2024-03-04T16:23:43.104991+0000","last_clean":"2024-03-04T16:23:43.104991+0000","last_became_active":"2024-03-04T16:21:25.449029+0000","last_became_peered":"2024-03-04T16:21:25.449029+0000","last_unstale":"2024-03-04T16:23:43.104991+0000","last_undegraded":"2024-03-04T16:23:43.104991+0000","last_fullsized":"2024-03-04T16:23:43.104991+0000","mapping_epoch":401,"log_start":"434'4200","ondisk_log_start":"434'4200","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:45.125902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272826368,"num_objects":4163,"num_object_clones":0,"num_object_copies":12489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9175,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":857,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937980+0000","last_change":"2024-03-04T16:23:04.261493+0000","last_active":"2024-03-04T16:23:37.937980+0000","last_peered":"2024-03-04T16:23:37.937980+0000","last_clean":"2024-03-04T16:23:37.937980+0000","last_became_active":"2024-03-04T16:21:45.424279+0000","last_became_peered":"2024-03-04T16:21:45.424279+0000","last_unstale":"2024-03-04T16:23:37.937980+0000","last_undegraded":"2024-03-04T16:23:37.937980+0000","last_fullsized":"2024-03-04T16:23:37.937980+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.261358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.261358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.709711+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":900,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930409+0000","last_change":"2024-03-04T16:21:07.575776+0000","last_active":"2024-03-04T16:23:37.930409+0000","last_peered":"2024-03-04T16:23:37.930409+0000","last_clean":"2024-03-04T16:23:37.930409+0000","last_became_active":"2024-03-04T16:20:07.493115+0000","last_became_peered":"2024-03-04T16:20:07.493115+0000","last_unstale":"2024-03-04T16:23:37.930409+0000","last_undegraded":"2024-03-04T16:23:37.930409+0000","last_fullsized":"2024-03-04T16:23:37.930409+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:21:07.575695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:07.575695+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.277602+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":974,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.935692+0000","last_change":"2024-03-04T16:23:00.064299+0000","last_active":"2024-03-04T16:23:37.935692+0000","last_peered":"2024-03-04T16:23:37.935692+0000","last_clean":"2024-03-04T16:23:37.935692+0000","last_became_active":"2024-03-04T16:21:45.422336+0000","last_became_peered":"2024-03-04T16:21:45.422336+0000","last_unstale":"2024-03-04T16:23:37.935692+0000","last_undegraded":"2024-03-04T16:23:37.935692+0000","last_fullsized":"2024-03-04T16:23:37.935692+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:23:00.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.064148+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.044607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"412'11","reported_seq":1142,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930267+0000","last_change":"2024-03-04T16:23:07.273194+0000","last_active":"2024-03-04T16:23:37.930267+0000","last_peered":"2024-03-04T16:23:37.930267+0000","last_clean":"2024-03-04T16:23:37.930267+0000","last_became_active":"2024-03-04T16:21:45.441048+0000","last_became_peered":"2024-03-04T16:21:45.441048+0000","last_unstale":"2024-03-04T16:23:37.930267+0000","last_undegraded":"2024-03-04T16:23:37.930267+0000","last_fullsized":"2024-03-04T16:23:37.930267+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'11","last_scrub_stamp":"2024-03-04T16:23:07.273057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.273057+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:35.024574+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"452'8748","reported_seq":8935,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.278733+0000","last_change":"2024-03-04T16:23:06.658833+0000","last_active":"2024-03-04T16:23:39.278733+0000","last_peered":"2024-03-04T16:23:39.278733+0000","last_clean":"2024-03-04T16:23:39.278733+0000","last_became_active":"2024-03-04T16:21:25.448720+0000","last_became_peered":"2024-03-04T16:21:25.448720+0000","last_unstale":"2024-03-04T16:23:39.278733+0000","last_undegraded":"2024-03-04T16:23:39.278733+0000","last_fullsized":"2024-03-04T16:23:39.278733+0000","mapping_epoch":401,"log_start":"434'4100","ondisk_log_start":"434'4100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6580","last_scrub_stamp":"2024-03-04T16:23:06.658440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.658440+0000","objects_scrubbed":6489,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.080402+0000","scrub_duration":3051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":13314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":824,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938010+0000","last_change":"2024-03-04T16:21:41.947473+0000","last_active":"2024-03-04T16:23:37.938010+0000","last_peered":"2024-03-04T16:23:37.938010+0000","last_clean":"2024-03-04T16:23:37.938010+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:23:37.938010+0000","last_undegraded":"2024-03-04T16:23:37.938010+0000","last_fullsized":"2024-03-04T16:23:37.938010+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:41.947350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.947350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074206+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":866,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930339+0000","last_change":"2024-03-04T16:22:50.302556+0000","last_active":"2024-03-04T16:23:37.930339+0000","last_peered":"2024-03-04T16:23:37.930339+0000","last_clean":"2024-03-04T16:23:37.930339+0000","last_became_active":"2024-03-04T16:20:07.497627+0000","last_became_peered":"2024-03-04T16:20:07.497627+0000","last_unstale":"2024-03-04T16:23:37.930339+0000","last_undegraded":"2024-03-04T16:23:37.930339+0000","last_fullsized":"2024-03-04T16:23:37.930339+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.302482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.302482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.673861+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":659,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939738+0000","last_change":"2024-03-04T16:22:52.266342+0000","last_active":"2024-03-04T16:23:37.939738+0000","last_peered":"2024-03-04T16:23:37.939738+0000","last_clean":"2024-03-04T16:23:37.939738+0000","last_became_active":"2024-03-04T16:21:45.443501+0000","last_became_peered":"2024-03-04T16:21:45.443501+0000","last_unstale":"2024-03-04T16:23:37.939738+0000","last_undegraded":"2024-03-04T16:23:37.939738+0000","last_fullsized":"2024-03-04T16:23:37.939738+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:22:52.266216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.266216+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.312618+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":249,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.928484+0000","last_change":"2024-03-04T16:22:02.701368+0000","last_active":"2024-03-04T16:23:37.928484+0000","last_peered":"2024-03-04T16:23:37.928484+0000","last_clean":"2024-03-04T16:23:37.928484+0000","last_became_active":"2024-03-04T16:19:04.768239+0000","last_became_peered":"2024-03-04T16:19:04.768239+0000","last_unstale":"2024-03-04T16:23:37.928484+0000","last_undegraded":"2024-03-04T16:23:37.928484+0000","last_fullsized":"2024-03-04T16:23:37.928484+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:02.701238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:02.701238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.215172+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"452'9237","reported_seq":11416,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.779680+0000","last_change":"2024-03-04T16:21:59.472159+0000","last_active":"2024-03-04T16:23:42.779680+0000","last_peered":"2024-03-04T16:23:42.779680+0000","last_clean":"2024-03-04T16:23:42.779680+0000","last_became_active":"2024-03-04T16:21:59.469208+0000","last_became_peered":"2024-03-04T16:21:59.469208+0000","last_unstale":"2024-03-04T16:23:42.779680+0000","last_undegraded":"2024-03-04T16:23:42.779680+0000","last_fullsized":"2024-03-04T16:23:42.779680+0000","mapping_epoch":421,"log_start":"435'4500","ondisk_log_start":"435'4500","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278462464,"num_objects":4249,"num_object_clones":0,"num_object_copies":12747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":431552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":67305472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":858,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929611+0000","last_change":"2024-03-04T16:21:44.902269+0000","last_active":"2024-03-04T16:23:37.929611+0000","last_peered":"2024-03-04T16:23:37.929611+0000","last_clean":"2024-03-04T16:23:37.929611+0000","last_became_active":"2024-03-04T16:19:04.749223+0000","last_became_peered":"2024-03-04T16:19:04.749223+0000","last_unstale":"2024-03-04T16:23:37.929611+0000","last_undegraded":"2024-03-04T16:23:37.929611+0000","last_fullsized":"2024-03-04T16:23:37.929611+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.902188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.902188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":249,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929470+0000","last_change":"2024-03-04T16:22:04.717449+0000","last_active":"2024-03-04T16:23:37.929470+0000","last_peered":"2024-03-04T16:23:37.929470+0000","last_clean":"2024-03-04T16:23:37.929470+0000","last_became_active":"2024-03-04T16:19:04.780445+0000","last_became_peered":"2024-03-04T16:19:04.780445+0000","last_unstale":"2024-03-04T16:23:37.929470+0000","last_undegraded":"2024-03-04T16:23:37.929470+0000","last_fullsized":"2024-03-04T16:23:37.929470+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:04.717368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.717368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.215206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":864,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930702+0000","last_change":"2024-03-04T16:21:59.968201+0000","last_active":"2024-03-04T16:23:37.930702+0000","last_peered":"2024-03-04T16:23:37.930702+0000","last_clean":"2024-03-04T16:23:37.930702+0000","last_became_active":"2024-03-04T16:21:45.411738+0000","last_became_peered":"2024-03-04T16:21:45.411738+0000","last_unstale":"2024-03-04T16:23:37.930702+0000","last_undegraded":"2024-03-04T16:23:37.930702+0000","last_fullsized":"2024-03-04T16:23:37.930702+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:59.968123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:21:59.968123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'52","reported_seq":175,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932314+0000","last_change":"2024-03-04T16:22:51.690727+0000","last_active":"2024-03-04T16:23:37.932314+0000","last_peered":"2024-03-04T16:23:37.932314+0000","last_clean":"2024-03-04T16:23:37.932314+0000","last_became_active":"2024-03-04T16:22:24.826414+0000","last_became_peered":"2024-03-04T16:22:24.826414+0000","last_unstale":"2024-03-04T16:23:37.932314+0000","last_undegraded":"2024-03-04T16:23:37.932314+0000","last_fullsized":"2024-03-04T16:23:37.932314+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:22:51.690576+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.690576+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:05.795083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"452'9239","reported_seq":11405,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.813686+0000","last_change":"2024-03-04T16:21:52.427898+0000","last_active":"2024-03-04T16:23:42.813686+0000","last_peered":"2024-03-04T16:23:42.813686+0000","last_clean":"2024-03-04T16:23:42.813686+0000","last_became_active":"2024-03-04T16:21:52.426535+0000","last_became_peered":"2024-03-04T16:21:52.426535+0000","last_unstale":"2024-03-04T16:23:42.813686+0000","last_undegraded":"2024-03-04T16:23:42.813686+0000","last_fullsized":"2024-03-04T16:23:42.813686+0000","mapping_epoch":414,"log_start":"435'4500","ondisk_log_start":"435'4500","created":401,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4739,"log_dups_size":0,"ondisk_log_size":4739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280428544,"num_objects":4279,"num_object_clones":0,"num_object_copies":12837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9239,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":66977792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":733,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929723+0000","last_change":"2024-03-04T16:22:06.237495+0000","last_active":"2024-03-04T16:23:37.929723+0000","last_peered":"2024-03-04T16:23:37.929723+0000","last_clean":"2024-03-04T16:23:37.929723+0000","last_became_active":"2024-03-04T16:19:04.779570+0000","last_became_peered":"2024-03-04T16:19:04.779570+0000","last_unstale":"2024-03-04T16:23:37.929723+0000","last_undegraded":"2024-03-04T16:23:37.929723+0000","last_fullsized":"2024-03-04T16:23:37.929723+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.237403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.237403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.279292+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":263,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932866+0000","last_change":"2024-03-04T16:22:54.512261+0000","last_active":"2024-03-04T16:23:37.932866+0000","last_peered":"2024-03-04T16:23:37.932866+0000","last_clean":"2024-03-04T16:23:37.932866+0000","last_became_active":"2024-03-04T16:20:07.487240+0000","last_became_peered":"2024-03-04T16:20:07.487240+0000","last_unstale":"2024-03-04T16:23:37.932866+0000","last_undegraded":"2024-03-04T16:23:37.932866+0000","last_fullsized":"2024-03-04T16:23:37.932866+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.512143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.512143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:01.584413+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":863,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936993+0000","last_change":"2024-03-04T16:23:03.094410+0000","last_active":"2024-03-04T16:23:37.936993+0000","last_peered":"2024-03-04T16:23:37.936993+0000","last_clean":"2024-03-04T16:23:37.936993+0000","last_became_active":"2024-03-04T16:21:45.422860+0000","last_became_peered":"2024-03-04T16:21:45.422860+0000","last_unstale":"2024-03-04T16:23:37.936993+0000","last_undegraded":"2024-03-04T16:23:37.936993+0000","last_fullsized":"2024-03-04T16:23:37.936993+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.094190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.094190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.574130+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"441'25","reported_seq":756,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932971+0000","last_change":"2024-03-04T16:22:56.668460+0000","last_active":"2024-03-04T16:23:37.932971+0000","last_peered":"2024-03-04T16:23:37.932971+0000","last_clean":"2024-03-04T16:23:37.932971+0000","last_became_active":"2024-03-04T16:21:10.564447+0000","last_became_peered":"2024-03-04T16:21:10.564447+0000","last_unstale":"2024-03-04T16:23:37.932971+0000","last_undegraded":"2024-03-04T16:23:37.932971+0000","last_fullsized":"2024-03-04T16:23:37.932971+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:22:56.668180+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.668180+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.974798+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"452'9012","reported_seq":9184,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.415040+0000","last_change":"2024-03-04T16:22:42.339861+0000","last_active":"2024-03-04T16:23:39.415040+0000","last_peered":"2024-03-04T16:23:39.415040+0000","last_clean":"2024-03-04T16:23:39.415040+0000","last_became_active":"2024-03-04T16:21:25.449702+0000","last_became_peered":"2024-03-04T16:21:25.449702+0000","last_unstale":"2024-03-04T16:23:39.415040+0000","last_undegraded":"2024-03-04T16:23:39.415040+0000","last_fullsized":"2024-03-04T16:23:39.415040+0000","mapping_epoch":401,"log_start":"435'4500","ondisk_log_start":"435'4500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5208","last_scrub_stamp":"2024-03-04T16:22:42.339680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.339680+0000","objects_scrubbed":5130,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.727729+0000","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288096256,"num_objects":4396,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9012,"num_write_kb":429056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":836,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938649+0000","last_change":"2024-03-04T16:22:52.913981+0000","last_active":"2024-03-04T16:23:37.938649+0000","last_peered":"2024-03-04T16:23:37.938649+0000","last_clean":"2024-03-04T16:23:37.938649+0000","last_became_active":"2024-03-04T16:21:45.423440+0000","last_became_peered":"2024-03-04T16:21:45.423440+0000","last_unstale":"2024-03-04T16:23:37.938649+0000","last_undegraded":"2024-03-04T16:23:37.938649+0000","last_fullsized":"2024-03-04T16:23:37.938649+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:52.913842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.913842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.335302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":875,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937830+0000","last_change":"2024-03-04T16:21:45.446577+0000","last_active":"2024-03-04T16:23:37.937830+0000","last_peered":"2024-03-04T16:23:37.937830+0000","last_clean":"2024-03-04T16:23:37.937830+0000","last_became_active":"2024-03-04T16:21:45.425232+0000","last_became_peered":"2024-03-04T16:21:45.425232+0000","last_unstale":"2024-03-04T16:23:37.937830+0000","last_undegraded":"2024-03-04T16:23:37.937830+0000","last_fullsized":"2024-03-04T16:23:37.937830+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:20:52.438469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:20:52.438469+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093024+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932009+0000","last_change":"2024-03-04T16:22:45.604898+0000","last_active":"2024-03-04T16:23:37.932009+0000","last_peered":"2024-03-04T16:23:37.932009+0000","last_clean":"2024-03-04T16:23:37.932009+0000","last_became_active":"2024-03-04T16:21:45.437679+0000","last_became_peered":"2024-03-04T16:21:45.437679+0000","last_unstale":"2024-03-04T16:23:37.932009+0000","last_undegraded":"2024-03-04T16:23:37.932009+0000","last_fullsized":"2024-03-04T16:23:37.932009+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.604769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.604769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.372412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"441'55","reported_seq":1269,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.050465+0000","last_change":"2024-03-04T16:22:04.783731+0000","last_active":"2024-03-04T16:23:39.050465+0000","last_peered":"2024-03-04T16:23:39.050465+0000","last_clean":"2024-03-04T16:23:39.050465+0000","last_became_active":"2024-03-04T16:19:04.769086+0000","last_became_peered":"2024-03-04T16:19:04.769086+0000","last_unstale":"2024-03-04T16:23:39.050465+0000","last_undegraded":"2024-03-04T16:23:39.050465+0000","last_fullsized":"2024-03-04T16:23:39.050465+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'43","last_scrub_stamp":"2024-03-04T16:22:04.783619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.783619+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.394129+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":700,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.933244+0000","last_change":"2024-03-04T16:21:45.434819+0000","last_active":"2024-03-04T16:23:37.933244+0000","last_peered":"2024-03-04T16:23:37.933244+0000","last_clean":"2024-03-04T16:23:37.933244+0000","last_became_active":"2024-03-04T16:21:45.434502+0000","last_became_peered":"2024-03-04T16:21:45.434502+0000","last_unstale":"2024-03-04T16:23:37.933244+0000","last_undegraded":"2024-03-04T16:23:37.933244+0000","last_fullsized":"2024-03-04T16:23:37.933244+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.410657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.410657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"452'9128","reported_seq":11363,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.496272+0000","last_change":"2024-03-04T16:22:04.820773+0000","last_active":"2024-03-04T16:23:42.496272+0000","last_peered":"2024-03-04T16:23:42.496272+0000","last_clean":"2024-03-04T16:23:42.496272+0000","last_became_active":"2024-03-04T16:22:04.786993+0000","last_became_peered":"2024-03-04T16:22:04.786993+0000","last_unstale":"2024-03-04T16:23:42.496272+0000","last_undegraded":"2024-03-04T16:23:42.496272+0000","last_fullsized":"2024-03-04T16:23:42.496272+0000","mapping_epoch":424,"log_start":"434'4200","ondisk_log_start":"434'4200","created":401,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272105472,"num_objects":4152,"num_object_clones":0,"num_object_copies":12456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4152,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9128,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":69074944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":859,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930893+0000","last_change":"2024-03-04T16:21:45.415808+0000","last_active":"2024-03-04T16:23:37.930893+0000","last_peered":"2024-03-04T16:23:37.930893+0000","last_clean":"2024-03-04T16:23:37.930893+0000","last_became_active":"2024-03-04T16:21:45.415473+0000","last_became_peered":"2024-03-04T16:21:45.415473+0000","last_unstale":"2024-03-04T16:23:37.930893+0000","last_undegraded":"2024-03-04T16:23:37.930893+0000","last_fullsized":"2024-03-04T16:23:37.930893+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:40.973478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.973478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":877,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940505+0000","last_change":"2024-03-04T16:22:51.222795+0000","last_active":"2024-03-04T16:23:37.940505+0000","last_peered":"2024-03-04T16:23:37.940505+0000","last_clean":"2024-03-04T16:23:37.940505+0000","last_became_active":"2024-03-04T16:21:45.437084+0000","last_became_peered":"2024-03-04T16:21:45.437084+0000","last_unstale":"2024-03-04T16:23:37.940505+0000","last_undegraded":"2024-03-04T16:23:37.940505+0000","last_fullsized":"2024-03-04T16:23:37.940505+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:51.222677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.222677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.054940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":257,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.928999+0000","last_change":"2024-03-04T16:22:09.686124+0000","last_active":"2024-03-04T16:23:37.928999+0000","last_peered":"2024-03-04T16:23:37.928999+0000","last_clean":"2024-03-04T16:23:37.928999+0000","last_became_active":"2024-03-04T16:19:04.780845+0000","last_became_peered":"2024-03-04T16:19:04.780845+0000","last_unstale":"2024-03-04T16:23:37.928999+0000","last_undegraded":"2024-03-04T16:23:37.928999+0000","last_fullsized":"2024-03-04T16:23:37.928999+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.685989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.685989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.053256+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":502,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.948240+0000","last_change":"2024-03-04T16:23:08.368485+0000","last_active":"2024-03-04T16:23:37.948240+0000","last_peered":"2024-03-04T16:23:37.948240+0000","last_clean":"2024-03-04T16:23:37.948240+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:23:37.948240+0000","last_undegraded":"2024-03-04T16:23:37.948240+0000","last_fullsized":"2024-03-04T16:23:37.948240+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-04T16:23:08.368095+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.368095+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.910733+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":257,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929883+0000","last_change":"2024-03-04T16:22:43.622761+0000","last_active":"2024-03-04T16:23:37.929883+0000","last_peered":"2024-03-04T16:23:37.929883+0000","last_clean":"2024-03-04T16:23:37.929883+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:23:37.929883+0000","last_undegraded":"2024-03-04T16:23:37.929883+0000","last_fullsized":"2024-03-04T16:23:37.929883+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.622383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.622383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.378061+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"452'8992","reported_seq":11084,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.594355+0000","last_change":"2024-03-04T16:21:58.471552+0000","last_active":"2024-03-04T16:23:42.594355+0000","last_peered":"2024-03-04T16:23:42.594355+0000","last_clean":"2024-03-04T16:23:42.594355+0000","last_became_active":"2024-03-04T16:21:58.469058+0000","last_became_peered":"2024-03-04T16:21:58.469058+0000","last_unstale":"2024-03-04T16:23:42.594355+0000","last_undegraded":"2024-03-04T16:23:42.594355+0000","last_fullsized":"2024-03-04T16:23:42.594355+0000","mapping_epoch":420,"log_start":"434'4100","ondisk_log_start":"434'4100","created":401,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281018368,"num_objects":4288,"num_object_clones":0,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":64487424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":845,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937831+0000","last_change":"2024-03-04T16:22:50.355727+0000","last_active":"2024-03-04T16:23:37.937831+0000","last_peered":"2024-03-04T16:23:37.937831+0000","last_clean":"2024-03-04T16:23:37.937831+0000","last_became_active":"2024-03-04T16:20:07.500640+0000","last_became_peered":"2024-03-04T16:20:07.500640+0000","last_unstale":"2024-03-04T16:23:37.937831+0000","last_undegraded":"2024-03-04T16:23:37.937831+0000","last_fullsized":"2024-03-04T16:23:37.937831+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.355598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.355598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.959365+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1175,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929705+0000","last_change":"2024-03-04T16:22:48.307804+0000","last_active":"2024-03-04T16:23:37.929705+0000","last_peered":"2024-03-04T16:23:37.929705+0000","last_clean":"2024-03-04T16:23:37.929705+0000","last_became_active":"2024-03-04T16:21:48.434375+0000","last_became_peered":"2024-03-04T16:21:48.434375+0000","last_unstale":"2024-03-04T16:23:37.929705+0000","last_undegraded":"2024-03-04T16:23:37.929705+0000","last_fullsized":"2024-03-04T16:23:37.929705+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:22:48.307675+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.307675+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.653556+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":775,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937671+0000","last_change":"2024-03-04T16:21:45.425510+0000","last_active":"2024-03-04T16:23:37.937671+0000","last_peered":"2024-03-04T16:23:37.937671+0000","last_clean":"2024-03-04T16:23:37.937671+0000","last_became_active":"2024-03-04T16:21:45.424708+0000","last_became_peered":"2024-03-04T16:21:45.424708+0000","last_unstale":"2024-03-04T16:23:37.937671+0000","last_undegraded":"2024-03-04T16:23:37.937671+0000","last_fullsized":"2024-03-04T16:23:37.937671+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:03.041001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:03.041001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093043+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"441'51","reported_seq":156,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.933713+0000","last_change":"2024-03-04T16:21:45.448127+0000","last_active":"2024-03-04T16:23:37.933713+0000","last_peered":"2024-03-04T16:23:37.933713+0000","last_clean":"2024-03-04T16:23:37.933713+0000","last_became_active":"2024-03-04T16:21:45.411357+0000","last_became_peered":"2024-03-04T16:21:45.411357+0000","last_unstale":"2024-03-04T16:23:37.933713+0000","last_undegraded":"2024-03-04T16:23:37.933713+0000","last_fullsized":"2024-03-04T16:23:37.933713+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'38","last_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_deep_scrub":"391'38","last_deep_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_clean_scrub_stamp":"2024-03-04T16:21:02.538605+0000","objects_scrubbed":8,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":865,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930586+0000","last_change":"2024-03-04T16:22:04.052615+0000","last_active":"2024-03-04T16:23:37.930586+0000","last_peered":"2024-03-04T16:23:37.930586+0000","last_clean":"2024-03-04T16:23:37.930586+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:23:37.930586+0000","last_undegraded":"2024-03-04T16:23:37.930586+0000","last_fullsized":"2024-03-04T16:23:37.930586+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:04.052531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.052531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"452'8868","reported_seq":11096,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.421656+0000","last_change":"2024-03-04T16:21:55.410964+0000","last_active":"2024-03-04T16:23:39.421656+0000","last_peered":"2024-03-04T16:23:39.421656+0000","last_clean":"2024-03-04T16:23:39.421656+0000","last_became_active":"2024-03-04T16:21:55.408063+0000","last_became_peered":"2024-03-04T16:21:55.408063+0000","last_unstale":"2024-03-04T16:23:39.421656+0000","last_undegraded":"2024-03-04T16:23:39.421656+0000","last_fullsized":"2024-03-04T16:23:39.421656+0000","mapping_epoch":417,"log_start":"434'4300","ondisk_log_start":"434'4300","created":401,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4568,"log_dups_size":0,"ondisk_log_size":4568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8868,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":69009408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":848,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940839+0000","last_change":"2024-03-04T16:23:03.390202+0000","last_active":"2024-03-04T16:23:37.940839+0000","last_peered":"2024-03-04T16:23:37.940839+0000","last_clean":"2024-03-04T16:23:37.940839+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:23:37.940839+0000","last_undegraded":"2024-03-04T16:23:37.940839+0000","last_fullsized":"2024-03-04T16:23:37.940839+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.389804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.389804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.921181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":702,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.933211+0000","last_change":"2024-03-04T16:21:38.419849+0000","last_active":"2024-03-04T16:23:37.933211+0000","last_peered":"2024-03-04T16:23:37.933211+0000","last_clean":"2024-03-04T16:23:37.933211+0000","last_became_active":"2024-03-04T16:20:07.498576+0000","last_became_peered":"2024-03-04T16:20:07.498576+0000","last_unstale":"2024-03-04T16:23:37.933211+0000","last_undegraded":"2024-03-04T16:23:37.933211+0000","last_fullsized":"2024-03-04T16:23:37.933211+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:38.419741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.419741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656836+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":2125,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:40.880657+0000","last_change":"2024-03-04T16:21:41.881246+0000","last_active":"2024-03-04T16:23:40.880657+0000","last_peered":"2024-03-04T16:23:40.880657+0000","last_clean":"2024-03-04T16:23:40.880657+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:23:40.880657+0000","last_undegraded":"2024-03-04T16:23:40.880657+0000","last_fullsized":"2024-03-04T16:23:40.880657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:21:41.881125+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.881125+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1251,"num_read_kb":1246,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":798,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.934237+0000","last_change":"2024-03-04T16:22:24.868227+0000","last_active":"2024-03-04T16:23:37.934237+0000","last_peered":"2024-03-04T16:23:37.934237+0000","last_clean":"2024-03-04T16:23:37.934237+0000","last_became_active":"2024-03-04T16:22:24.838293+0000","last_became_peered":"2024-03-04T16:22:24.838293+0000","last_unstale":"2024-03-04T16:23:37.934237+0000","last_undegraded":"2024-03-04T16:23:37.934237+0000","last_fullsized":"2024-03-04T16:23:37.934237+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:22:09.593105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.593105+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:39.656866+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":851,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938310+0000","last_change":"2024-03-04T16:22:45.128630+0000","last_active":"2024-03-04T16:23:37.938310+0000","last_peered":"2024-03-04T16:23:37.938310+0000","last_clean":"2024-03-04T16:23:37.938310+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:23:37.938310+0000","last_undegraded":"2024-03-04T16:23:37.938310+0000","last_fullsized":"2024-03-04T16:23:37.938310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.128548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.128548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:46.395177+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"452'8882","reported_seq":9033,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.351750+0000","last_change":"2024-03-04T16:23:12.943073+0000","last_active":"2024-03-04T16:23:39.351750+0000","last_peered":"2024-03-04T16:23:39.351750+0000","last_clean":"2024-03-04T16:23:39.351750+0000","last_became_active":"2024-03-04T16:21:25.448277+0000","last_became_peered":"2024-03-04T16:21:25.448277+0000","last_unstale":"2024-03-04T16:23:39.351750+0000","last_undegraded":"2024-03-04T16:23:39.351750+0000","last_fullsized":"2024-03-04T16:23:39.351750+0000","mapping_epoch":401,"log_start":"436'4600","ondisk_log_start":"436'4600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":3799,"log_size":4282,"log_dups_size":0,"ondisk_log_size":4282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.393917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288358400,"num_objects":4400,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8882,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":864,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939614+0000","last_change":"2024-03-04T16:22:02.934144+0000","last_active":"2024-03-04T16:23:37.939614+0000","last_peered":"2024-03-04T16:23:37.939614+0000","last_clean":"2024-03-04T16:23:37.939614+0000","last_became_active":"2024-03-04T16:21:45.442233+0000","last_became_peered":"2024-03-04T16:21:45.442233+0000","last_unstale":"2024-03-04T16:23:37.939614+0000","last_undegraded":"2024-03-04T16:23:37.939614+0000","last_fullsized":"2024-03-04T16:23:37.939614+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:02.934010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_clean_scrub_stamp":"2024-03-04T16:22:02.934010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.387004+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":857,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.941146+0000","last_change":"2024-03-04T16:22:48.112648+0000","last_active":"2024-03-04T16:23:37.941146+0000","last_peered":"2024-03-04T16:23:37.941146+0000","last_clean":"2024-03-04T16:23:37.941146+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:23:37.941146+0000","last_undegraded":"2024-03-04T16:23:37.941146+0000","last_fullsized":"2024-03-04T16:23:37.941146+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.112512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.112512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.233790+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'47","reported_seq":841,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936541+0000","last_change":"2024-03-04T16:22:49.132475+0000","last_active":"2024-03-04T16:23:37.936541+0000","last_peered":"2024-03-04T16:23:37.936541+0000","last_clean":"2024-03-04T16:23:37.936541+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:23:37.936541+0000","last_undegraded":"2024-03-04T16:23:37.936541+0000","last_fullsized":"2024-03-04T16:23:37.936541+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:22:49.132304+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:22:49.132304+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.744012+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932110+0000","last_change":"2024-03-04T16:22:08.771819+0000","last_active":"2024-03-04T16:23:37.932110+0000","last_peered":"2024-03-04T16:23:37.932110+0000","last_clean":"2024-03-04T16:23:37.932110+0000","last_became_active":"2024-03-04T16:21:45.412152+0000","last_became_peered":"2024-03-04T16:21:45.412152+0000","last_unstale":"2024-03-04T16:23:37.932110+0000","last_undegraded":"2024-03-04T16:23:37.932110+0000","last_fullsized":"2024-03-04T16:23:37.932110+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_clean_scrub_stamp":"2024-03-04T16:22:08.770293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"452'9163","reported_seq":11415,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:43.091513+0000","last_change":"2024-03-04T16:22:54.725063+0000","last_active":"2024-03-04T16:23:43.091513+0000","last_peered":"2024-03-04T16:23:43.091513+0000","last_clean":"2024-03-04T16:23:43.091513+0000","last_became_active":"2024-03-04T16:21:54.483830+0000","last_became_peered":"2024-03-04T16:21:54.483830+0000","last_unstale":"2024-03-04T16:23:43.091513+0000","last_undegraded":"2024-03-04T16:23:43.091513+0000","last_fullsized":"2024-03-04T16:23:43.091513+0000","mapping_epoch":416,"log_start":"429'3600","ondisk_log_start":"429'3600","created":401,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'5897","last_scrub_stamp":"2024-03-04T16:22:54.724570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.724570+0000","objects_scrubbed":5842,"log_size":5563,"log_dups_size":0,"ondisk_log_size":5563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.844871+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270139416,"num_objects":4123,"num_object_clones":0,"num_object_copies":12369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4123,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9163,"num_write_kb":425089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":861,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932418+0000","last_change":"2024-03-04T16:22:47.720482+0000","last_active":"2024-03-04T16:23:37.932418+0000","last_peered":"2024-03-04T16:23:37.932418+0000","last_clean":"2024-03-04T16:23:37.932418+0000","last_became_active":"2024-03-04T16:19:04.834912+0000","last_became_peered":"2024-03-04T16:19:04.834912+0000","last_unstale":"2024-03-04T16:23:37.932418+0000","last_undegraded":"2024-03-04T16:23:37.932418+0000","last_fullsized":"2024-03-04T16:23:37.932418+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.720278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.720278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.926758+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":862,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936980+0000","last_change":"2024-03-04T16:20:51.445972+0000","last_active":"2024-03-04T16:23:37.936980+0000","last_peered":"2024-03-04T16:23:37.936980+0000","last_clean":"2024-03-04T16:23:37.936980+0000","last_became_active":"2024-03-04T16:19:04.762915+0000","last_became_peered":"2024-03-04T16:19:04.762915+0000","last_unstale":"2024-03-04T16:23:37.936980+0000","last_undegraded":"2024-03-04T16:23:37.936980+0000","last_fullsized":"2024-03-04T16:23:37.936980+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_clean_scrub_stamp":"2024-03-04T16:20:51.445890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:40.093090+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":917,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.940146+0000","last_change":"2024-03-04T16:22:00.856182+0000","last_active":"2024-03-04T16:23:37.940146+0000","last_peered":"2024-03-04T16:23:37.940146+0000","last_clean":"2024-03-04T16:23:37.940146+0000","last_became_active":"2024-03-04T16:20:07.504270+0000","last_became_peered":"2024-03-04T16:20:07.504270+0000","last_unstale":"2024-03-04T16:23:37.940146+0000","last_undegraded":"2024-03-04T16:23:37.940146+0000","last_fullsized":"2024-03-04T16:23:37.940146+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:22:00.856053+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:22:00.856053+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.394099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"412'12","reported_seq":132,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932149+0000","last_change":"2024-03-04T16:23:09.792730+0000","last_active":"2024-03-04T16:23:37.932149+0000","last_peered":"2024-03-04T16:23:37.932149+0000","last_clean":"2024-03-04T16:23:37.932149+0000","last_became_active":"2024-03-04T16:21:45.414139+0000","last_became_peered":"2024-03-04T16:21:45.414139+0000","last_unstale":"2024-03-04T16:23:37.932149+0000","last_undegraded":"2024-03-04T16:23:37.932149+0000","last_fullsized":"2024-03-04T16:23:37.932149+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12","last_scrub_stamp":"2024-03-04T16:23:09.792648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.792648+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:09.926315+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"452'8990","reported_seq":9176,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.372635+0000","last_change":"2024-03-04T16:23:12.469661+0000","last_active":"2024-03-04T16:23:39.372635+0000","last_peered":"2024-03-04T16:23:39.372635+0000","last_clean":"2024-03-04T16:23:39.372635+0000","last_became_active":"2024-03-04T16:21:25.451006+0000","last_became_peered":"2024-03-04T16:21:25.451006+0000","last_unstale":"2024-03-04T16:23:39.372635+0000","last_undegraded":"2024-03-04T16:23:39.372635+0000","last_fullsized":"2024-03-04T16:23:39.372635+0000","mapping_epoch":401,"log_start":"437'4700","ondisk_log_start":"437'4700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7132","last_scrub_stamp":"2024-03-04T16:23:12.469418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.469418+0000","objects_scrubbed":6443,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.303121+0000","scrub_duration":3140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8990,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":873,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.931090+0000","last_change":"2024-03-04T16:23:10.418223+0000","last_active":"2024-03-04T16:23:37.931090+0000","last_peered":"2024-03-04T16:23:37.931090+0000","last_clean":"2024-03-04T16:23:37.931090+0000","last_became_active":"2024-03-04T16:21:45.413068+0000","last_became_peered":"2024-03-04T16:21:45.413068+0000","last_unstale":"2024-03-04T16:23:37.931090+0000","last_undegraded":"2024-03-04T16:23:37.931090+0000","last_fullsized":"2024-03-04T16:23:37.931090+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:10.418117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.418117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.395028+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":710,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.934410+0000","last_change":"2024-03-04T16:22:53.683732+0000","last_active":"2024-03-04T16:23:37.934410+0000","last_peered":"2024-03-04T16:23:37.934410+0000","last_clean":"2024-03-04T16:23:37.934410+0000","last_became_active":"2024-03-04T16:20:07.494182+0000","last_became_peered":"2024-03-04T16:20:07.494182+0000","last_unstale":"2024-03-04T16:23:37.934410+0000","last_undegraded":"2024-03-04T16:23:37.934410+0000","last_fullsized":"2024-03-04T16:23:37.934410+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:53.683136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:53.683136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.670431+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'49","reported_seq":710,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936012+0000","last_change":"2024-03-04T16:23:07.122889+0000","last_active":"2024-03-04T16:23:37.936012+0000","last_peered":"2024-03-04T16:23:37.936012+0000","last_clean":"2024-03-04T16:23:37.936012+0000","last_became_active":"2024-03-04T16:21:45.423250+0000","last_became_peered":"2024-03-04T16:21:45.423250+0000","last_unstale":"2024-03-04T16:23:37.936012+0000","last_undegraded":"2024-03-04T16:23:37.936012+0000","last_fullsized":"2024-03-04T16:23:37.936012+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'49","last_scrub_stamp":"2024-03-04T16:23:07.122781+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.122781+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.026776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"366'9","reported_seq":1099,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939023+0000","last_change":"2024-03-04T16:22:08.721644+0000","last_active":"2024-03-04T16:23:37.939023+0000","last_peered":"2024-03-04T16:23:37.939023+0000","last_clean":"2024-03-04T16:23:37.939023+0000","last_became_active":"2024-03-04T16:20:07.484919+0000","last_became_peered":"2024-03-04T16:20:07.484919+0000","last_unstale":"2024-03-04T16:23:37.939023+0000","last_undegraded":"2024-03-04T16:23:37.939023+0000","last_fullsized":"2024-03-04T16:23:37.939023+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:22:08.721229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:08.721229+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.074236+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"452'9291","reported_seq":9404,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.804956+0000","last_change":"2024-03-04T16:21:25.449756+0000","last_active":"2024-03-04T16:23:42.804956+0000","last_peered":"2024-03-04T16:23:42.804956+0000","last_clean":"2024-03-04T16:23:42.804956+0000","last_became_active":"2024-03-04T16:21:25.449469+0000","last_became_peered":"2024-03-04T16:21:25.449469+0000","last_unstale":"2024-03-04T16:23:42.804956+0000","last_undegraded":"2024-03-04T16:23:42.804956+0000","last_fullsized":"2024-03-04T16:23:42.804956+0000","mapping_epoch":401,"log_start":"435'4600","ondisk_log_start":"435'4600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4691,"log_dups_size":0,"ondisk_log_size":4691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279904256,"num_objects":4271,"num_object_clones":0,"num_object_copies":12813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9291,"num_write_kb":433984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":804,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.946709+0000","last_change":"2024-03-04T16:23:04.878400+0000","last_active":"2024-03-04T16:23:37.946709+0000","last_peered":"2024-03-04T16:23:37.946709+0000","last_clean":"2024-03-04T16:23:37.946709+0000","last_became_active":"2024-03-04T16:21:45.424326+0000","last_became_peered":"2024-03-04T16:21:45.424326+0000","last_unstale":"2024-03-04T16:23:37.946709+0000","last_undegraded":"2024-03-04T16:23:37.946709+0000","last_fullsized":"2024-03-04T16:23:37.946709+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.878278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.878278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:30.211352+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":307,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930313+0000","last_change":"2024-03-04T16:22:42.336778+0000","last_active":"2024-03-04T16:23:37.930313+0000","last_peered":"2024-03-04T16:23:37.930313+0000","last_clean":"2024-03-04T16:23:37.930313+0000","last_became_active":"2024-03-04T16:21:25.438797+0000","last_became_peered":"2024-03-04T16:21:25.438797+0000","last_unstale":"2024-03-04T16:23:37.930313+0000","last_undegraded":"2024-03-04T16:23:37.930313+0000","last_fullsized":"2024-03-04T16:23:37.930313+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:42.336656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.336656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.043125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"441'42","reported_seq":957,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938839+0000","last_change":"2024-03-04T16:23:07.898360+0000","last_active":"2024-03-04T16:23:37.938839+0000","last_peered":"2024-03-04T16:23:37.938839+0000","last_clean":"2024-03-04T16:23:37.938839+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:23:37.938839+0000","last_undegraded":"2024-03-04T16:23:37.938839+0000","last_fullsized":"2024-03-04T16:23:37.938839+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-04T16:23:07.898070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.898070+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:32.268466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":195,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938244+0000","last_change":"2024-03-04T16:22:43.866317+0000","last_active":"2024-03-04T16:23:37.938244+0000","last_peered":"2024-03-04T16:23:37.938244+0000","last_clean":"2024-03-04T16:23:37.938244+0000","last_became_active":"2024-03-04T16:20:07.487202+0000","last_became_peered":"2024-03-04T16:20:07.487202+0000","last_unstale":"2024-03-04T16:23:37.938244+0000","last_undegraded":"2024-03-04T16:23:37.938244+0000","last_fullsized":"2024-03-04T16:23:37.938244+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.866200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.866200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:45.649412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"452'8796","reported_seq":8968,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:39.344572+0000","last_change":"2024-03-04T16:22:42.067136+0000","last_active":"2024-03-04T16:23:39.344572+0000","last_peered":"2024-03-04T16:23:39.344572+0000","last_clean":"2024-03-04T16:23:39.344572+0000","last_became_active":"2024-03-04T16:21:25.450165+0000","last_became_peered":"2024-03-04T16:21:25.450165+0000","last_unstale":"2024-03-04T16:23:39.344572+0000","last_undegraded":"2024-03-04T16:23:39.344572+0000","last_fullsized":"2024-03-04T16:23:39.344572+0000","mapping_epoch":401,"log_start":"435'4500","ondisk_log_start":"435'4500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5014","last_scrub_stamp":"2024-03-04T16:22:42.066893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.066893+0000","objects_scrubbed":4968,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.270207+0000","scrub_duration":2144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292028416,"num_objects":4456,"num_object_clones":0,"num_object_copies":13368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8796,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":843,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.936691+0000","last_change":"2024-03-04T16:22:46.243647+0000","last_active":"2024-03-04T16:23:37.936691+0000","last_peered":"2024-03-04T16:23:37.936691+0000","last_clean":"2024-03-04T16:23:37.936691+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:23:37.936691+0000","last_undegraded":"2024-03-04T16:23:37.936691+0000","last_fullsized":"2024-03-04T16:23:37.936691+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.243413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.243413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:51.507147+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"366'9","reported_seq":267,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937519+0000","last_change":"2024-03-04T16:23:02.057648+0000","last_active":"2024-03-04T16:23:37.937519+0000","last_peered":"2024-03-04T16:23:37.937519+0000","last_clean":"2024-03-04T16:23:37.937519+0000","last_became_active":"2024-03-04T16:20:07.484334+0000","last_became_peered":"2024-03-04T16:20:07.484334+0000","last_unstale":"2024-03-04T16:23:37.937519+0000","last_undegraded":"2024-03-04T16:23:37.937519+0000","last_fullsized":"2024-03-04T16:23:37.937519+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:23:02.057434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.057434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.009367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":844,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938752+0000","last_change":"2024-03-04T16:22:48.920668+0000","last_active":"2024-03-04T16:23:37.938752+0000","last_peered":"2024-03-04T16:23:37.938752+0000","last_clean":"2024-03-04T16:23:37.938752+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:23:37.938752+0000","last_undegraded":"2024-03-04T16:23:37.938752+0000","last_fullsized":"2024-03-04T16:23:37.938752+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.920550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.920550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.681425+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'46","reported_seq":938,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.948349+0000","last_change":"2024-03-04T16:23:00.358198+0000","last_active":"2024-03-04T16:23:37.948349+0000","last_peered":"2024-03-04T16:23:37.948349+0000","last_clean":"2024-03-04T16:23:37.948349+0000","last_became_active":"2024-03-04T16:20:07.507258+0000","last_became_peered":"2024-03-04T16:20:07.507258+0000","last_unstale":"2024-03-04T16:23:37.948349+0000","last_undegraded":"2024-03-04T16:23:37.948349+0000","last_fullsized":"2024-03-04T16:23:37.948349+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:23:00.358056+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.358056+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.797848+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"452'9254","reported_seq":11180,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.584111+0000","last_change":"2024-03-04T16:21:56.410088+0000","last_active":"2024-03-04T16:23:42.584111+0000","last_peered":"2024-03-04T16:23:42.584111+0000","last_clean":"2024-03-04T16:23:42.584111+0000","last_became_active":"2024-03-04T16:21:56.407370+0000","last_became_peered":"2024-03-04T16:21:56.407370+0000","last_unstale":"2024-03-04T16:23:42.584111+0000","last_undegraded":"2024-03-04T16:23:42.584111+0000","last_fullsized":"2024-03-04T16:23:42.584111+0000","mapping_epoch":418,"log_start":"434'4400","ondisk_log_start":"434'4400","created":401,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280625152,"num_objects":4282,"num_object_clones":0,"num_object_copies":12846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9254,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1068,"num_bytes_recovered":69992448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":109,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932307+0000","last_change":"2024-03-04T16:23:00.740759+0000","last_active":"2024-03-04T16:23:37.932307+0000","last_peered":"2024-03-04T16:23:37.932307+0000","last_clean":"2024-03-04T16:23:37.932307+0000","last_became_active":"2024-03-04T16:21:45.420505+0000","last_became_peered":"2024-03-04T16:21:45.420505+0000","last_unstale":"2024-03-04T16:23:37.932307+0000","last_undegraded":"2024-03-04T16:23:37.932307+0000","last_fullsized":"2024-03-04T16:23:37.932307+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:23:00.740631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.740631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.137776+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":867,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.938223+0000","last_change":"2024-03-04T16:23:12.207020+0000","last_active":"2024-03-04T16:23:37.938223+0000","last_peered":"2024-03-04T16:23:37.938223+0000","last_clean":"2024-03-04T16:23:37.938223+0000","last_became_active":"2024-03-04T16:21:45.425391+0000","last_became_peered":"2024-03-04T16:21:45.425391+0000","last_unstale":"2024-03-04T16:23:37.938223+0000","last_undegraded":"2024-03-04T16:23:37.938223+0000","last_fullsized":"2024-03-04T16:23:37.938223+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.206943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.206943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:38.403715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":804,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929094+0000","last_change":"2024-03-04T16:22:06.694118+0000","last_active":"2024-03-04T16:23:37.929094+0000","last_peered":"2024-03-04T16:23:37.929094+0000","last_clean":"2024-03-04T16:23:37.929094+0000","last_became_active":"2024-03-04T16:20:07.471591+0000","last_became_peered":"2024-03-04T16:20:07.471591+0000","last_unstale":"2024-03-04T16:23:37.929094+0000","last_undegraded":"2024-03-04T16:23:37.929094+0000","last_fullsized":"2024-03-04T16:23:37.929094+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.693989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.693989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.215273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"441'32","reported_seq":129,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932569+0000","last_change":"2024-03-04T16:23:05.801666+0000","last_active":"2024-03-04T16:23:37.932569+0000","last_peered":"2024-03-04T16:23:37.932569+0000","last_clean":"2024-03-04T16:23:37.932569+0000","last_became_active":"2024-03-04T16:21:45.446219+0000","last_became_peered":"2024-03-04T16:21:45.446219+0000","last_unstale":"2024-03-04T16:23:37.932569+0000","last_undegraded":"2024-03-04T16:23:37.932569+0000","last_fullsized":"2024-03-04T16:23:37.932569+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:23:05.801531+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:23:05.801531+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.742131+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"452'9353","reported_seq":9539,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:43.023980+0000","last_change":"2024-03-04T16:22:58.382258+0000","last_active":"2024-03-04T16:23:43.023980+0000","last_peered":"2024-03-04T16:23:43.023980+0000","last_clean":"2024-03-04T16:23:43.023980+0000","last_became_active":"2024-03-04T16:21:25.445028+0000","last_became_peered":"2024-03-04T16:21:25.445028+0000","last_unstale":"2024-03-04T16:23:43.023980+0000","last_undegraded":"2024-03-04T16:23:43.023980+0000","last_fullsized":"2024-03-04T16:23:43.023980+0000","mapping_epoch":401,"log_start":"430'3700","ondisk_log_start":"430'3700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6281","last_scrub_stamp":"2024-03-04T16:22:58.381858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.381858+0000","objects_scrubbed":6184,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.638463+0000","scrub_duration":2894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279248896,"num_objects":4261,"num_object_clones":0,"num_object_copies":12783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":269,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.930926+0000","last_change":"2024-03-04T16:22:10.868223+0000","last_active":"2024-03-04T16:23:37.930926+0000","last_peered":"2024-03-04T16:23:37.930926+0000","last_clean":"2024-03-04T16:23:37.930926+0000","last_became_active":"2024-03-04T16:21:45.423031+0000","last_became_peered":"2024-03-04T16:21:45.423031+0000","last_unstale":"2024-03-04T16:23:37.930926+0000","last_undegraded":"2024-03-04T16:23:37.930926+0000","last_fullsized":"2024-03-04T16:23:37.930926+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:10.867960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:10.867960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:42.311500+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":856,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.941391+0000","last_change":"2024-03-04T16:21:46.008458+0000","last_active":"2024-03-04T16:23:37.941391+0000","last_peered":"2024-03-04T16:23:37.941391+0000","last_clean":"2024-03-04T16:23:37.941391+0000","last_became_active":"2024-03-04T16:21:45.434815+0000","last_became_peered":"2024-03-04T16:21:45.434815+0000","last_unstale":"2024-03-04T16:23:37.941391+0000","last_undegraded":"2024-03-04T16:23:37.941391+0000","last_fullsized":"2024-03-04T16:23:37.941391+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:46.008376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:46.008376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:38.386898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":105,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.932370+0000","last_change":"2024-03-04T16:23:01.769342+0000","last_active":"2024-03-04T16:23:37.932370+0000","last_peered":"2024-03-04T16:23:37.932370+0000","last_clean":"2024-03-04T16:23:37.932370+0000","last_became_active":"2024-03-04T16:21:45.423438+0000","last_became_peered":"2024-03-04T16:21:45.423438+0000","last_unstale":"2024-03-04T16:23:37.932370+0000","last_undegraded":"2024-03-04T16:23:37.932370+0000","last_fullsized":"2024-03-04T16:23:37.932370+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:01.769222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:23:01.769222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:05.705541+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":962,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929801+0000","last_change":"2024-03-04T16:23:10.470966+0000","last_active":"2024-03-04T16:23:37.929801+0000","last_peered":"2024-03-04T16:23:37.929801+0000","last_clean":"2024-03-04T16:23:37.929801+0000","last_became_active":"2024-03-04T16:21:45.439301+0000","last_became_peered":"2024-03-04T16:21:45.439301+0000","last_unstale":"2024-03-04T16:23:37.929801+0000","last_undegraded":"2024-03-04T16:23:37.929801+0000","last_fullsized":"2024-03-04T16:23:37.929801+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-04T16:23:10.470699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.470699+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.738180+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"452'9280","reported_seq":11017,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:42.587178+0000","last_change":"2024-03-04T16:21:59.474717+0000","last_active":"2024-03-04T16:23:42.587178+0000","last_peered":"2024-03-04T16:23:42.587178+0000","last_clean":"2024-03-04T16:23:42.587178+0000","last_became_active":"2024-03-04T16:21:59.471792+0000","last_became_peered":"2024-03-04T16:21:59.471792+0000","last_unstale":"2024-03-04T16:23:42.587178+0000","last_undegraded":"2024-03-04T16:23:42.587178+0000","last_fullsized":"2024-03-04T16:23:42.587178+0000","mapping_epoch":421,"log_start":"434'4400","ondisk_log_start":"434'4400","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4880,"log_dups_size":0,"ondisk_log_size":4880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:43.106453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278134784,"num_objects":4244,"num_object_clones":0,"num_object_copies":12732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9280,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":70123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":792,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.937115+0000","last_change":"2024-03-04T16:22:44.238092+0000","last_active":"2024-03-04T16:23:37.937115+0000","last_peered":"2024-03-04T16:23:37.937115+0000","last_clean":"2024-03-04T16:23:37.937115+0000","last_became_active":"2024-03-04T16:20:07.480213+0000","last_became_peered":"2024-03-04T16:20:07.480213+0000","last_unstale":"2024-03-04T16:23:37.937115+0000","last_undegraded":"2024-03-04T16:23:37.937115+0000","last_fullsized":"2024-03-04T16:23:37.937115+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.237641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.237641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.682755+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":317,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:38.070650+0000","last_change":"2024-03-04T16:22:58.485899+0000","last_active":"2024-03-04T16:23:38.070650+0000","last_peered":"2024-03-04T16:23:38.070650+0000","last_clean":"2024-03-04T16:23:38.070650+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:23:38.070650+0000","last_undegraded":"2024-03-04T16:23:38.070650+0000","last_fullsized":"2024-03-04T16:23:38.070650+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:58.485608+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.485608+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.537438+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":130,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.929012+0000","last_change":"2024-03-04T16:23:12.437298+0000","last_active":"2024-03-04T16:23:37.929012+0000","last_peered":"2024-03-04T16:23:37.929012+0000","last_clean":"2024-03-04T16:23:37.929012+0000","last_became_active":"2024-03-04T16:21:24.444615+0000","last_became_peered":"2024-03-04T16:21:24.444615+0000","last_unstale":"2024-03-04T16:23:37.929012+0000","last_undegraded":"2024-03-04T16:23:37.929012+0000","last_fullsized":"2024-03-04T16:23:37.929012+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.437183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.437183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.188381+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":844,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-04T16:23:37.939161+0000","last_change":"2024-03-04T16:22:56.926525+0000","last_active":"2024-03-04T16:23:37.939161+0000","last_peered":"2024-03-04T16:23:37.939161+0000","last_clean":"2024-03-04T16:23:37.939161+0000","last_became_active":"2024-03-04T16:21:45.425802+0000","last_became_peered":"2024-03-04T16:21:45.425802+0000","last_unstale":"2024-03-04T16:23:37.939161+0000","last_undegraded":"2024-03-04T16:23:37.939161+0000","last_fullsized":"2024-03-04T16:23:37.939161+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:22:56.926373+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.926373+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.476141+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4513136664,"num_objects":68866,"num_object_clones":0,"num_object_copies":206598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68866,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145408,"num_write_kb":6856705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8309,"num_bytes_recovered":544538624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13614657536,"data_stored":13614645320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76508,"ondisk_log_size":76508,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1251,"num_read_kb":1246,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":8906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070720,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2830784,"kb_used_data":1915256,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":90914368,"statfs":{"total":95995035648,"available":93096312832,"internally_reserved":0,"allocated":1961222144,"data_stored":1960808740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":937490890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365060,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2565440,"kb_used_data":2191352,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":91179712,"statfs":{"total":95995035648,"available":93368025088,"internally_reserved":0,"allocated":2243944448,"data_stored":2243529594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3664860,"kb_used_data":2560468,"kb_used_omap":1,"kb_used_meta":1104318,"kb_avail":90080292,"statfs":{"total":95995035648,"available":92242219008,"internally_reserved":0,"allocated":2621919232,"data_stored":2621475812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1130822085},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2925376,"kb_used_data":2019572,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":90819776,"statfs":{"total":95995035648,"available":92999450624,"internally_reserved":0,"allocated":2068041728,"data_stored":2067631762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411608,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2224360,"kb_used_data":1637728,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91520792,"statfs":{"total":95995035648,"available":93717291008,"internally_reserved":0,"allocated":1677033472,"data_stored":1676642486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149970,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3179212,"kb_used_data":2007812,"kb_used_omap":1,"kb_used_meta":1171326,"kb_avail":90565940,"statfs":{"total":95995035648,"available":92739522560,"internally_reserved":0,"allocated":2055999488,"data_stored":2055572770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1199438280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626698,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2116204,"kb_used_data":1092388,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":91628948,"statfs":{"total":95995035648,"available":93828042752,"internally_reserved":0,"allocated":1118605312,"data_stored":1118210032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1048377798},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2046754816,"data_stored":2046754816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2619543552,"data_stored":2619539480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1651380224,"data_stored":1651376152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2041053184,"data_stored":2041053184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1097596928,"data_stored":1097596928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2217349120,"data_stored":2217345048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1940979712,"data_stored":1940979712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:23:47.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:23:47.158 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:47.214 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:47.316 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:47.344 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:47.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:47.446 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:47.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:47.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:47.623 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:47.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:47.725 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:47.750 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:47.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:47.897 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:47.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:47.999 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:48.029 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:48.132 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:48.158 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:48.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:48.286 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:48.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:48.415 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:48.493 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:23:48.494 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:23:48.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.563 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:48.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:48.664 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:48.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:48.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:48.842 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:48.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:48.992 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:48.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[29779]: pgmap v988: 148 pgs: 148 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:23:48.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:23:48.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:48.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[38113]: pgmap v988: 148 pgs: 148 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:48.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:23:48.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:23:48.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:48 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:49.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:49.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:48 smithi176 ceph-mon[33619]: pgmap v988: 148 pgs: 148 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-04T16:23:49.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:48 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:23:49.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:48 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:23:49.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:48 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:23:49.122 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:49.224 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.252 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:49.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:49.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.410 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:49.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:49.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:49.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.555 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:49.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:49.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:49.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:49.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.852 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:49.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:49.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:49.997 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:49.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:50.098 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:50.129 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:50.232 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.272 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:50.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:50.373 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:50.403 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:50.505 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:50.531 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:50.633 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.686 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:50.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:50.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:50.816 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:50.844 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:23:50.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:50.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:23:50.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:50.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:50.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:51.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:51.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:51.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:50 smithi176 ceph-mon[33619]: pgmap v989: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 865 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:51.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:50 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4276048693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:50 smithi176 ceph-mon[33619]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.206 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:51.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[38113]: pgmap v989: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 865 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:51.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4276048693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[38113]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[29779]: pgmap v989: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 865 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4276048693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:50 smithi038 ceph-mon[29779]: from='client.25906 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:23:51.333 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:51.360 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:51.462 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:51.595 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:51.627 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:51.729 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:51.758 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:51.860 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:51.889 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:51.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:51.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:51.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[29779]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:23:51.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[29779]: osdmap e453: 8 total, 8 up, 7 in 2024-03-04T16:23:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[38113]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:23:51.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:51 smithi038 ceph-mon[38113]: osdmap e453: 8 total, 8 up, 7 in 2024-03-04T16:23:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.036 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:52.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:52.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:51 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:23:52.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:51 smithi176 ceph-mon[33619]: from='client.25906 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:23:52.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:51 smithi176 ceph-mon[33619]: osdmap e453: 8 total, 8 up, 7 in 2024-03-04T16:23:52.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:52.167 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:52.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:52.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:52.407 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:52.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:52.537 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:52.564 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:52.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:52.666 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:52.694 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:52.796 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:52.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.830 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:52.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:52.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:52.966 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:52.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:53.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:53.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:53.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:53.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:52 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:23:53.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:52 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:23:53.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:52 smithi176 ceph-mon[33619]: pgmap v991: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:53.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:52 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:23:53.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:52 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:23:53.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:53.229 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:53.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:23:53.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:23:53.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[38113]: pgmap v991: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[29779]: pgmap v991: 148 pgs: 148 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:23:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:52 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:23:53.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.391 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:53.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:53.492 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:53.524 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:53.604 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:23:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:23:53.626 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:53.652 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:53.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:53.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:53.756 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:53.837 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:53.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:23:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:23:53.938 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:53.990 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:53.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:54.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.137 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:54.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:54.182 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[29779]: 11.c scrub starts 2024-03-04T16:23:54.182 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:54.182 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:54.182 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:54.183 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:23:54.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[38113]: 11.c scrub starts 2024-03-04T16:23:54.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:54.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:54.183 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:53 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:54.224 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:23:54.238 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:54.265 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:54.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:23:54.315 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:23:54.354 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:23:54.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:53 smithi176 ceph-mon[33619]: 11.c scrub starts 2024-03-04T16:23:54.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:53 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:54.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:53 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:54.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:53 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:23:54.367 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:54.397 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:23:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.429 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:54.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:54.433 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:23:54.481 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:23:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:23:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:23:54.530 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:54.557 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:54.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:54.659 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:54.706 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:54.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:54.835 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:54.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:54.937 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:54.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:54.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: pgmap v992: 148 pgs: 148 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: osdmap e454: 8 total, 8 up, 7 in 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: 11.c scrub ok 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:23:54.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/968693598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: pgmap v992: 148 pgs: 148 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: osdmap e454: 8 total, 8 up, 7 in 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: 11.c scrub ok 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:23:54.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/968693598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:55.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:55.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:55.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:55.230 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:55.320 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:23:55.332 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:55.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:23:55.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: pgmap v992: 148 pgs: 148 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:23:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: osdmap e454: 8 total, 8 up, 7 in 2024-03-04T16:23:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: 11.c scrub ok 2024-03-04T16:23:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:23:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:23:55.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/968693598' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:23:55.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:55.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:55.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.489 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:55.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:55.590 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:55.619 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:55.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:55.760 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:55.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:55.900 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:55.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:56.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:56.028 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:56.130 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:56.159 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[29779]: 11.9 scrub starts 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:23:56.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[38113]: 11.9 scrub starts 2024-03-04T16:23:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:23:56.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:55 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:23:56.258 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:56.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:56.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:55 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:23:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:55 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:23:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:55 smithi176 ceph-mon[33619]: 11.9 scrub starts 2024-03-04T16:23:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:55 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:23:56.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:55 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:23:56.389 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:56.417 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:56.519 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:56.546 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:56.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:56.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.681 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:56.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:56.811 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:56.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:56.955 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:56.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:57.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:57.086 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:57.188 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:57.196 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: pgmap v994: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:57.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[29779]: osdmap e455: 8 total, 8 up, 7 in 2024-03-04T16:23:57.198 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:23:57.198 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:23:57.198 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:23:57.199 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:23:57.199 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: pgmap v994: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:57.199 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:57.199 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:57.199 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:56 smithi038 ceph-mon[38113]: osdmap e455: 8 total, 8 up, 7 in 2024-03-04T16:23:57.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.221 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:57.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:57.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: pgmap v994: 148 pgs: 148 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:23:57.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:56 smithi176 ceph-mon[33619]: osdmap e455: 8 total, 8 up, 7 in 2024-03-04T16:23:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:57.464 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:57.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:57.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:57.634 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:57.736 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:57.882 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:57.908 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:57.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:57.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[29779]: 11.9 scrub ok 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[38113]: 11.9 scrub ok 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:23:57.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:57 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:23:58.011 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:58.041 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:58.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:58.143 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.177 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:58.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:58.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:23:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.324 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:23:58.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:57 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:23:58.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:57 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:23:58.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:57 smithi176 ceph-mon[33619]: 11.9 scrub ok 2024-03-04T16:23:58.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:57 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:23:58.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:57 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:23:58.425 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:58.454 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:58.556 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:23:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:58.615 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:23:58.629 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:23:58.705 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:58.735 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:58.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:58.837 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:58.865 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:58.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:58.967 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:59.011 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:59.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:59.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:23:59.113 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:23:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.142 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:59.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 11.a scrub starts 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: 11.a scrub ok 2024-03-04T16:23:59.149 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[29779]: pgmap v996: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 980 op/s 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 11.a scrub starts 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: 11.a scrub ok 2024-03-04T16:23:59.150 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:58 smithi038 ceph-mon[38113]: pgmap v996: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 980 op/s 2024-03-04T16:23:59.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:23:59.277 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:23:59.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 11.a scrub starts 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: 11.a scrub ok 2024-03-04T16:23:59.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:58 smithi176 ceph-mon[33619]: pgmap v996: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 980 op/s 2024-03-04T16:23:59.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:23:59.406 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:23:59.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:23:59.550 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:23:59.652 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:23:59.683 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:23:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:23:59.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:23:59.823 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:23:59.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:23:59.963 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:23:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:23:59.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:23:59.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:23:59.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:23:59.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: from='client.35990 v1:172.21.15.38:0/27119184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: osdmap e456: 8 total, 8 up, 7 in 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: from='client.35990 v1:172.21.15.38:0/27119184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: osdmap e456: 8 total, 8 up, 7 in 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:23:59.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:23:59 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:24:00.066 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:00.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:00.163 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":996,"stamp":"2024-03-04T16:23:57.757237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3577485686,"num_objects":54778,"num_object_clones":0,"num_object_copies":164334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54778,"num_whiteouts":0,"num_read":3028,"num_read_kb":2874,"num_write":160881,"num_write_kb":6859166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8805,"num_bytes_recovered":548685156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77678,"ondisk_log_size":77678,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16755000,"kb_used_data":10627572,"kb_used_omap":10,"kb_used_meta":6127157,"kb_avail":639461064,"statfs":{"total":671965249536,"available":654808129536,"internally_reserved":0,"allocated":10882633728,"data_stored":10879746180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":6274208877},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-443940864,"num_objects":-6774,"num_object_clones":0,"num_object_copies":-20322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6774,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6774,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.916630"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":197,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.833985+0000","last_change":"2024-03-04T16:21:44.712801+0000","last_active":"2024-03-04T16:23:50.833985+0000","last_peered":"2024-03-04T16:23:50.833985+0000","last_clean":"2024-03-04T16:23:50.833985+0000","last_became_active":"2024-03-04T16:20:07.476108+0000","last_became_peered":"2024-03-04T16:20:07.476108+0000","last_unstale":"2024-03-04T16:23:50.833985+0000","last_undegraded":"2024-03-04T16:23:50.833985+0000","last_fullsized":"2024-03-04T16:23:50.833985+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.712721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.712721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.079896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875345+0000","last_change":"2024-03-04T16:21:45.018101+0000","last_active":"2024-03-04T16:23:53.875345+0000","last_peered":"2024-03-04T16:23:53.875345+0000","last_clean":"2024-03-04T16:23:53.875345+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:23:53.875345+0000","last_undegraded":"2024-03-04T16:23:53.875345+0000","last_fullsized":"2024-03-04T16:23:53.875345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:45.018020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:21:45.018020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":710,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874053+0000","last_change":"2024-03-04T16:22:06.502649+0000","last_active":"2024-03-04T16:23:53.874053+0000","last_peered":"2024-03-04T16:23:53.874053+0000","last_clean":"2024-03-04T16:23:53.874053+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:23:53.874053+0000","last_undegraded":"2024-03-04T16:23:53.874053+0000","last_fullsized":"2024-03-04T16:23:53.874053+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.502545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.502545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.575076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":122,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.477593+0000","last_change":"2024-03-04T16:21:45.470673+0000","last_active":"2024-03-04T16:23:56.477593+0000","last_peered":"2024-03-04T16:23:56.477593+0000","last_clean":"2024-03-04T16:23:56.477593+0000","last_became_active":"2024-03-04T16:21:45.423494+0000","last_became_peered":"2024-03-04T16:21:45.423494+0000","last_unstale":"2024-03-04T16:23:56.477593+0000","last_undegraded":"2024-03-04T16:23:56.477593+0000","last_fullsized":"2024-03-04T16:23:56.477593+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:21:38.860021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.860021+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":839,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.867113+0000","last_change":"2024-03-04T16:21:45.429466+0000","last_active":"2024-03-04T16:23:53.867113+0000","last_peered":"2024-03-04T16:23:53.867113+0000","last_clean":"2024-03-04T16:23:53.867113+0000","last_became_active":"2024-03-04T16:21:45.427819+0000","last_became_peered":"2024-03-04T16:21:45.427819+0000","last_unstale":"2024-03-04T16:23:53.867113+0000","last_undegraded":"2024-03-04T16:23:53.867113+0000","last_fullsized":"2024-03-04T16:23:53.867113+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.058061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.058061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048707+0000","scrub_duration":301,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":111,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.477749+0000","last_change":"2024-03-04T16:22:56.684863+0000","last_active":"2024-03-04T16:23:56.477749+0000","last_peered":"2024-03-04T16:23:56.477749+0000","last_clean":"2024-03-04T16:23:56.477749+0000","last_became_active":"2024-03-04T16:21:45.422482+0000","last_became_peered":"2024-03-04T16:21:45.422482+0000","last_unstale":"2024-03-04T16:23:56.477749+0000","last_undegraded":"2024-03-04T16:23:56.477749+0000","last_fullsized":"2024-03-04T16:23:56.477749+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.684724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.684724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.493825+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":865,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.864938+0000","last_change":"2024-03-04T16:23:08.925901+0000","last_active":"2024-03-04T16:23:53.864938+0000","last_peered":"2024-03-04T16:23:53.864938+0000","last_clean":"2024-03-04T16:23:53.864938+0000","last_became_active":"2024-03-04T16:21:45.422756+0000","last_became_peered":"2024-03-04T16:21:45.422756+0000","last_unstale":"2024-03-04T16:23:53.864938+0000","last_undegraded":"2024-03-04T16:23:53.864938+0000","last_fullsized":"2024-03-04T16:23:53.864938+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.925783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.299467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'39","reported_seq":840,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.852384+0000","last_change":"2024-03-04T16:22:47.640816+0000","last_active":"2024-03-04T16:23:50.852384+0000","last_peered":"2024-03-04T16:23:50.852384+0000","last_clean":"2024-03-04T16:23:50.852384+0000","last_became_active":"2024-03-04T16:21:25.444226+0000","last_became_peered":"2024-03-04T16:21:25.444226+0000","last_unstale":"2024-03-04T16:23:50.852384+0000","last_undegraded":"2024-03-04T16:23:50.852384+0000","last_fullsized":"2024-03-04T16:23:50.852384+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:22:47.640638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.640638+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.818716+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":699,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870334+0000","last_change":"2024-03-04T16:22:50.786398+0000","last_active":"2024-03-04T16:23:53.870334+0000","last_peered":"2024-03-04T16:23:53.870334+0000","last_clean":"2024-03-04T16:23:53.870334+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:23:53.870334+0000","last_undegraded":"2024-03-04T16:23:53.870334+0000","last_fullsized":"2024-03-04T16:23:53.870334+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.786274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.786274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.345657+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":836,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834279+0000","last_change":"2024-03-04T16:22:06.888682+0000","last_active":"2024-03-04T16:23:50.834279+0000","last_peered":"2024-03-04T16:23:50.834279+0000","last_clean":"2024-03-04T16:23:50.834279+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:23:50.834279+0000","last_undegraded":"2024-03-04T16:23:50.834279+0000","last_fullsized":"2024-03-04T16:23:50.834279+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.888592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.079985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":870,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834974+0000","last_change":"2024-03-04T16:21:45.443994+0000","last_active":"2024-03-04T16:23:50.834974+0000","last_peered":"2024-03-04T16:23:50.834974+0000","last_clean":"2024-03-04T16:23:50.834974+0000","last_became_active":"2024-03-04T16:21:45.443239+0000","last_became_peered":"2024-03-04T16:21:45.443239+0000","last_unstale":"2024-03-04T16:23:50.834974+0000","last_undegraded":"2024-03-04T16:23:50.834974+0000","last_fullsized":"2024-03-04T16:23:50.834974+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:36.877837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:36.877837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.079924+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'25","reported_seq":664,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.869893+0000","last_change":"2024-03-04T16:22:09.726711+0000","last_active":"2024-03-04T16:23:53.869893+0000","last_peered":"2024-03-04T16:23:53.869893+0000","last_clean":"2024-03-04T16:23:53.869893+0000","last_became_active":"2024-03-04T16:20:07.491874+0000","last_became_peered":"2024-03-04T16:20:07.491874+0000","last_unstale":"2024-03-04T16:23:53.869893+0000","last_undegraded":"2024-03-04T16:23:53.869893+0000","last_fullsized":"2024-03-04T16:23:53.869893+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:09.726573+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.726573+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048847+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":840,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.833862+0000","last_change":"2024-03-04T16:20:56.455146+0000","last_active":"2024-03-04T16:23:50.833862+0000","last_peered":"2024-03-04T16:23:50.833862+0000","last_clean":"2024-03-04T16:23:50.833862+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:23:50.833862+0000","last_undegraded":"2024-03-04T16:23:50.833862+0000","last_fullsized":"2024-03-04T16:23:50.833862+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:56.455020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:20:56.455020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078133+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":859,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837131+0000","last_change":"2024-03-04T16:22:46.297643+0000","last_active":"2024-03-04T16:23:50.837131+0000","last_peered":"2024-03-04T16:23:50.837131+0000","last_clean":"2024-03-04T16:23:50.837131+0000","last_became_active":"2024-03-04T16:20:07.494164+0000","last_became_peered":"2024-03-04T16:20:07.494164+0000","last_unstale":"2024-03-04T16:23:50.837131+0000","last_undegraded":"2024-03-04T16:23:50.837131+0000","last_fullsized":"2024-03-04T16:23:50.837131+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.297535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.297535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.975907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":862,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834823+0000","last_change":"2024-03-04T16:21:45.422865+0000","last_active":"2024-03-04T16:23:50.834823+0000","last_peered":"2024-03-04T16:23:50.834823+0000","last_clean":"2024-03-04T16:23:50.834823+0000","last_became_active":"2024-03-04T16:21:45.422460+0000","last_became_peered":"2024-03-04T16:21:45.422460+0000","last_unstale":"2024-03-04T16:23:50.834823+0000","last_undegraded":"2024-03-04T16:23:50.834823+0000","last_fullsized":"2024-03-04T16:23:50.834823+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:55.405634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:20:55.405634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"441'11","reported_seq":632,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.866956+0000","last_change":"2024-03-04T16:23:00.985610+0000","last_active":"2024-03-04T16:23:53.866956+0000","last_peered":"2024-03-04T16:23:53.866956+0000","last_clean":"2024-03-04T16:23:53.866956+0000","last_became_active":"2024-03-04T16:20:07.482406+0000","last_became_peered":"2024-03-04T16:20:07.482406+0000","last_unstale":"2024-03-04T16:23:53.866956+0000","last_undegraded":"2024-03-04T16:23:53.866956+0000","last_fullsized":"2024-03-04T16:23:53.866956+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'11","last_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_deep_scrub":"441'11","last_deep_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.985229+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.484493+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":112,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.475760+0000","last_change":"2024-03-04T16:22:59.694873+0000","last_active":"2024-03-04T16:23:56.475760+0000","last_peered":"2024-03-04T16:23:56.475760+0000","last_clean":"2024-03-04T16:23:56.475760+0000","last_became_active":"2024-03-04T16:21:45.425723+0000","last_became_peered":"2024-03-04T16:21:45.425723+0000","last_unstale":"2024-03-04T16:23:56.475760+0000","last_undegraded":"2024-03-04T16:23:56.475760+0000","last_fullsized":"2024-03-04T16:23:56.475760+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:59.694757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:59.694757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:19.218070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":853,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876406+0000","last_change":"2024-03-04T16:21:41.913454+0000","last_active":"2024-03-04T16:23:53.876406+0000","last_peered":"2024-03-04T16:23:53.876406+0000","last_clean":"2024-03-04T16:23:53.876406+0000","last_became_active":"2024-03-04T16:20:07.497650+0000","last_became_peered":"2024-03-04T16:20:07.497650+0000","last_unstale":"2024-03-04T16:23:53.876406+0000","last_undegraded":"2024-03-04T16:23:53.876406+0000","last_fullsized":"2024-03-04T16:23:53.876406+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:41.913323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.913323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":697,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874089+0000","last_change":"2024-03-04T16:22:45.654818+0000","last_active":"2024-03-04T16:23:53.874089+0000","last_peered":"2024-03-04T16:23:53.874089+0000","last_clean":"2024-03-04T16:23:53.874089+0000","last_became_active":"2024-03-04T16:20:07.498539+0000","last_became_peered":"2024-03-04T16:20:07.498539+0000","last_unstale":"2024-03-04T16:23:53.874089+0000","last_undegraded":"2024-03-04T16:23:53.874089+0000","last_fullsized":"2024-03-04T16:23:53.874089+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.654354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.654354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.175429+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":284,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.848359+0000","last_change":"2024-03-04T16:21:55.824199+0000","last_active":"2024-03-04T16:23:50.848359+0000","last_peered":"2024-03-04T16:23:50.848359+0000","last_clean":"2024-03-04T16:23:50.848359+0000","last_became_active":"2024-03-04T16:19:04.771891+0000","last_became_peered":"2024-03-04T16:19:04.771891+0000","last_unstale":"2024-03-04T16:23:50.848359+0000","last_undegraded":"2024-03-04T16:23:50.848359+0000","last_fullsized":"2024-03-04T16:23:50.848359+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"286'34","last_scrub_stamp":"2024-03-04T16:21:55.824051+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:21:55.824051+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":112,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476297+0000","last_change":"2024-03-04T16:22:43.806087+0000","last_active":"2024-03-04T16:23:56.476297+0000","last_peered":"2024-03-04T16:23:56.476297+0000","last_clean":"2024-03-04T16:23:56.476297+0000","last_became_active":"2024-03-04T16:21:45.420331+0000","last_became_peered":"2024-03-04T16:21:45.420331+0000","last_unstale":"2024-03-04T16:23:56.476297+0000","last_undegraded":"2024-03-04T16:23:56.476297+0000","last_fullsized":"2024-03-04T16:23:56.476297+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.805945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.805945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.468231+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":799,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.864373+0000","last_change":"2024-03-04T16:21:42.722963+0000","last_active":"2024-03-04T16:23:53.864373+0000","last_peered":"2024-03-04T16:23:53.864373+0000","last_clean":"2024-03-04T16:23:53.864373+0000","last_became_active":"2024-03-04T16:20:07.474386+0000","last_became_peered":"2024-03-04T16:20:07.474386+0000","last_unstale":"2024-03-04T16:23:53.864373+0000","last_undegraded":"2024-03-04T16:23:53.864373+0000","last_fullsized":"2024-03-04T16:23:53.864373+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:42.722804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:42.722804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":815,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875466+0000","last_change":"2024-03-04T16:22:56.344093+0000","last_active":"2024-03-04T16:23:53.875466+0000","last_peered":"2024-03-04T16:23:53.875466+0000","last_clean":"2024-03-04T16:23:53.875466+0000","last_became_active":"2024-03-04T16:20:07.503831+0000","last_became_peered":"2024-03-04T16:20:07.503831+0000","last_unstale":"2024-03-04T16:23:53.875466+0000","last_undegraded":"2024-03-04T16:23:53.875466+0000","last_fullsized":"2024-03-04T16:23:53.875466+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.343792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.343792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.303419+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":560,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876172+0000","last_change":"2024-03-04T16:21:45.475840+0000","last_active":"2024-03-04T16:23:53.876172+0000","last_peered":"2024-03-04T16:23:53.876172+0000","last_clean":"2024-03-04T16:23:53.876172+0000","last_became_active":"2024-03-04T16:21:45.441225+0000","last_became_peered":"2024-03-04T16:21:45.441225+0000","last_unstale":"2024-03-04T16:23:53.876172+0000","last_undegraded":"2024-03-04T16:23:53.876172+0000","last_fullsized":"2024-03-04T16:23:53.876172+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'18","last_scrub_stamp":"2024-03-04T16:21:40.886581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.886581+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":863,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839614+0000","last_change":"2024-03-04T16:22:45.271732+0000","last_active":"2024-03-04T16:23:50.839614+0000","last_peered":"2024-03-04T16:23:50.839614+0000","last_clean":"2024-03-04T16:23:50.839614+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:23:50.839614+0000","last_undegraded":"2024-03-04T16:23:50.839614+0000","last_fullsized":"2024-03-04T16:23:50.839614+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:22:45.271595+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.271595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:56.454962+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"412'13","reported_seq":134,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476016+0000","last_change":"2024-03-04T16:21:45.443536+0000","last_active":"2024-03-04T16:23:56.476016+0000","last_peered":"2024-03-04T16:23:56.476016+0000","last_clean":"2024-03-04T16:23:56.476016+0000","last_became_active":"2024-03-04T16:21:45.416857+0000","last_became_peered":"2024-03-04T16:21:45.416857+0000","last_unstale":"2024-03-04T16:23:56.476016+0000","last_undegraded":"2024-03-04T16:23:56.476016+0000","last_fullsized":"2024-03-04T16:23:56.476016+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'12","last_scrub_stamp":"2024-03-04T16:21:38.843730+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.843730+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":876,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875589+0000","last_change":"2024-03-04T16:22:43.057543+0000","last_active":"2024-03-04T16:23:53.875589+0000","last_peered":"2024-03-04T16:23:53.875589+0000","last_clean":"2024-03-04T16:23:53.875589+0000","last_became_active":"2024-03-04T16:21:45.437760+0000","last_became_peered":"2024-03-04T16:21:45.437760+0000","last_unstale":"2024-03-04T16:23:53.875589+0000","last_undegraded":"2024-03-04T16:23:53.875589+0000","last_fullsized":"2024-03-04T16:23:53.875589+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:43.057419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.057419+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:01.275240+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":743,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874250+0000","last_change":"2024-03-04T16:23:02.578786+0000","last_active":"2024-03-04T16:23:53.874250+0000","last_peered":"2024-03-04T16:23:53.874250+0000","last_clean":"2024-03-04T16:23:53.874250+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:23:53.874250+0000","last_undegraded":"2024-03-04T16:23:53.874250+0000","last_fullsized":"2024-03-04T16:23:53.874250+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:23:02.578648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.578648+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.686651+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":849,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876550+0000","last_change":"2024-03-04T16:23:06.666597+0000","last_active":"2024-03-04T16:23:53.876550+0000","last_peered":"2024-03-04T16:23:53.876550+0000","last_clean":"2024-03-04T16:23:53.876550+0000","last_became_active":"2024-03-04T16:20:07.477199+0000","last_became_peered":"2024-03-04T16:20:07.477199+0000","last_unstale":"2024-03-04T16:23:53.876550+0000","last_undegraded":"2024-03-04T16:23:53.876550+0000","last_fullsized":"2024-03-04T16:23:53.876550+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:06.666475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:08.786009+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.666475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:33.618544+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":376,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.873800+0000","last_change":"2024-03-04T16:22:57.654317+0000","last_active":"2024-03-04T16:23:53.873800+0000","last_peered":"2024-03-04T16:23:53.873800+0000","last_clean":"2024-03-04T16:23:53.873800+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:23:53.873800+0000","last_undegraded":"2024-03-04T16:23:53.873800+0000","last_fullsized":"2024-03-04T16:23:53.873800+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:57.653816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.653816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.989031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":851,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.865901+0000","last_change":"2024-03-04T16:21:38.659764+0000","last_active":"2024-03-04T16:23:53.865901+0000","last_peered":"2024-03-04T16:23:53.865901+0000","last_clean":"2024-03-04T16:23:53.865901+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:23:53.865901+0000","last_undegraded":"2024-03-04T16:23:53.865901+0000","last_fullsized":"2024-03-04T16:23:53.865901+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:38.659652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.659652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'27","reported_seq":136,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476198+0000","last_change":"2024-03-04T16:22:42.533910+0000","last_active":"2024-03-04T16:23:56.476198+0000","last_peered":"2024-03-04T16:23:56.476198+0000","last_clean":"2024-03-04T16:23:56.476198+0000","last_became_active":"2024-03-04T16:21:45.434840+0000","last_became_peered":"2024-03-04T16:21:45.434840+0000","last_unstale":"2024-03-04T16:23:56.476198+0000","last_undegraded":"2024-03-04T16:23:56.476198+0000","last_fullsized":"2024-03-04T16:23:56.476198+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:42.533824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.533824+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":868,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837252+0000","last_change":"2024-03-04T16:22:09.161333+0000","last_active":"2024-03-04T16:23:50.837252+0000","last_peered":"2024-03-04T16:23:50.837252+0000","last_clean":"2024-03-04T16:23:50.837252+0000","last_became_active":"2024-03-04T16:20:07.477661+0000","last_became_peered":"2024-03-04T16:20:07.477661+0000","last_unstale":"2024-03-04T16:23:50.837252+0000","last_undegraded":"2024-03-04T16:23:50.837252+0000","last_fullsized":"2024-03-04T16:23:50.837252+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.161217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.161217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286434+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":111,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476459+0000","last_change":"2024-03-04T16:23:08.842921+0000","last_active":"2024-03-04T16:23:56.476459+0000","last_peered":"2024-03-04T16:23:56.476459+0000","last_clean":"2024-03-04T16:23:56.476459+0000","last_became_active":"2024-03-04T16:21:45.414027+0000","last_became_peered":"2024-03-04T16:21:45.414027+0000","last_unstale":"2024-03-04T16:23:56.476459+0000","last_undegraded":"2024-03-04T16:23:56.476459+0000","last_fullsized":"2024-03-04T16:23:56.476459+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.842842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.842842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:21.737638+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":691,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.873955+0000","last_change":"2024-03-04T16:22:10.615994+0000","last_active":"2024-03-04T16:23:53.873955+0000","last_peered":"2024-03-04T16:23:53.873955+0000","last_clean":"2024-03-04T16:23:53.873955+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:23:53.873955+0000","last_undegraded":"2024-03-04T16:23:53.873955+0000","last_fullsized":"2024-03-04T16:23:53.873955+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:10.615811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:10.615811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.575050+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"441'12","reported_seq":156,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.847451+0000","last_change":"2024-03-04T16:22:50.599109+0000","last_active":"2024-03-04T16:23:50.847451+0000","last_peered":"2024-03-04T16:23:50.847451+0000","last_clean":"2024-03-04T16:23:50.847451+0000","last_became_active":"2024-03-04T16:21:45.423994+0000","last_became_peered":"2024-03-04T16:21:45.423994+0000","last_unstale":"2024-03-04T16:23:50.847451+0000","last_undegraded":"2024-03-04T16:23:50.847451+0000","last_fullsized":"2024-03-04T16:23:50.847451+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:22:50.598967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.598967+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.026066+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":858,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.867327+0000","last_change":"2024-03-04T16:23:03.173401+0000","last_active":"2024-03-04T16:23:53.867327+0000","last_peered":"2024-03-04T16:23:53.867327+0000","last_clean":"2024-03-04T16:23:53.867327+0000","last_became_active":"2024-03-04T16:21:45.429844+0000","last_became_peered":"2024-03-04T16:21:45.429844+0000","last_unstale":"2024-03-04T16:23:53.867327+0000","last_undegraded":"2024-03-04T16:23:53.867327+0000","last_fullsized":"2024-03-04T16:23:53.867327+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.173174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.173174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.851084+0000","scrub_duration":244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":864,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839927+0000","last_change":"2024-03-04T16:22:54.333690+0000","last_active":"2024-03-04T16:23:50.839927+0000","last_peered":"2024-03-04T16:23:50.839927+0000","last_clean":"2024-03-04T16:23:50.839927+0000","last_became_active":"2024-03-04T16:21:45.415087+0000","last_became_peered":"2024-03-04T16:21:45.415087+0000","last_unstale":"2024-03-04T16:23:50.839927+0000","last_undegraded":"2024-03-04T16:23:50.839927+0000","last_fullsized":"2024-03-04T16:23:50.839927+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.333535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.333535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.663716+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":852,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835197+0000","last_change":"2024-03-04T16:22:41.028799+0000","last_active":"2024-03-04T16:23:50.835197+0000","last_peered":"2024-03-04T16:23:50.835197+0000","last_clean":"2024-03-04T16:23:50.835197+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:23:50.835197+0000","last_undegraded":"2024-03-04T16:23:50.835197+0000","last_fullsized":"2024-03-04T16:23:50.835197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.028680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.028680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.495192+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'27","reported_seq":915,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.867507+0000","last_change":"2024-03-04T16:22:55.927787+0000","last_active":"2024-03-04T16:23:53.867507+0000","last_peered":"2024-03-04T16:23:53.867507+0000","last_clean":"2024-03-04T16:23:53.867507+0000","last_became_active":"2024-03-04T16:21:45.428057+0000","last_became_peered":"2024-03-04T16:21:45.428057+0000","last_unstale":"2024-03-04T16:23:53.867507+0000","last_undegraded":"2024-03-04T16:23:53.867507+0000","last_fullsized":"2024-03-04T16:23:53.867507+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:22:55.927235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:55.927235+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.688732+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":691,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870495+0000","last_change":"2024-03-04T16:22:46.734503+0000","last_active":"2024-03-04T16:23:53.870495+0000","last_peered":"2024-03-04T16:23:53.870495+0000","last_clean":"2024-03-04T16:23:53.870495+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:23:53.870495+0000","last_undegraded":"2024-03-04T16:23:53.870495+0000","last_fullsized":"2024-03-04T16:23:53.870495+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.734392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.734392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:10.402153+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":479,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834189+0000","last_change":"2024-03-04T16:21:25.449367+0000","last_active":"2024-03-04T16:23:50.834189+0000","last_peered":"2024-03-04T16:23:50.834189+0000","last_clean":"2024-03-04T16:23:50.834189+0000","last_became_active":"2024-03-04T16:21:25.449034+0000","last_became_peered":"2024-03-04T16:21:25.449034+0000","last_unstale":"2024-03-04T16:23:50.834189+0000","last_undegraded":"2024-03-04T16:23:50.834189+0000","last_fullsized":"2024-03-04T16:23:50.834189+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:52.030278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:20:52.030278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":859,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837856+0000","last_change":"2024-03-04T16:21:58.944553+0000","last_active":"2024-03-04T16:23:50.837856+0000","last_peered":"2024-03-04T16:23:50.837856+0000","last_clean":"2024-03-04T16:23:50.837856+0000","last_became_active":"2024-03-04T16:20:07.498410+0000","last_became_peered":"2024-03-04T16:20:07.498410+0000","last_unstale":"2024-03-04T16:23:50.837856+0000","last_undegraded":"2024-03-04T16:23:50.837856+0000","last_fullsized":"2024-03-04T16:23:50.837856+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:58.944477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:58.944477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'25","reported_seq":86,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:52.014215+0000","last_change":"2024-03-04T16:23:52.014266+0000","last_active":"2024-03-04T16:23:52.014215+0000","last_peered":"2024-03-04T16:23:52.014215+0000","last_clean":"2024-03-04T16:23:52.014215+0000","last_became_active":"2024-03-04T16:22:24.825469+0000","last_became_peered":"2024-03-04T16:22:24.825469+0000","last_unstale":"2024-03-04T16:23:52.014215+0000","last_undegraded":"2024-03-04T16:23:52.014215+0000","last_fullsized":"2024-03-04T16:23:52.014215+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:23:52.014128+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:23:52.014128+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:14.597749+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":283,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870930+0000","last_change":"2024-03-04T16:22:44.719886+0000","last_active":"2024-03-04T16:23:53.870930+0000","last_peered":"2024-03-04T16:23:53.870930+0000","last_clean":"2024-03-04T16:23:53.870930+0000","last_became_active":"2024-03-04T16:21:25.446654+0000","last_became_peered":"2024-03-04T16:21:25.446654+0000","last_unstale":"2024-03-04T16:23:53.870930+0000","last_undegraded":"2024-03-04T16:23:53.870930+0000","last_fullsized":"2024-03-04T16:23:53.870930+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.719762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.719762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":848,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875739+0000","last_change":"2024-03-04T16:22:47.060545+0000","last_active":"2024-03-04T16:23:53.875739+0000","last_peered":"2024-03-04T16:23:53.875739+0000","last_clean":"2024-03-04T16:23:53.875739+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:23:53.875739+0000","last_undegraded":"2024-03-04T16:23:53.875739+0000","last_fullsized":"2024-03-04T16:23:53.875739+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.060423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.060423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:56.705586+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":811,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839401+0000","last_change":"2024-03-04T16:21:45.416172+0000","last_active":"2024-03-04T16:23:50.839401+0000","last_peered":"2024-03-04T16:23:50.839401+0000","last_clean":"2024-03-04T16:23:50.839401+0000","last_became_active":"2024-03-04T16:21:45.415525+0000","last_became_peered":"2024-03-04T16:21:45.415525+0000","last_unstale":"2024-03-04T16:23:50.839401+0000","last_undegraded":"2024-03-04T16:23:50.839401+0000","last_fullsized":"2024-03-04T16:23:50.839401+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:37.800150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:37.800150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284764+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":135,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.475932+0000","last_change":"2024-03-04T16:22:52.714265+0000","last_active":"2024-03-04T16:23:56.475932+0000","last_peered":"2024-03-04T16:23:56.475932+0000","last_clean":"2024-03-04T16:23:56.475932+0000","last_became_active":"2024-03-04T16:21:45.437462+0000","last_became_peered":"2024-03-04T16:21:45.437462+0000","last_unstale":"2024-03-04T16:23:56.475932+0000","last_undegraded":"2024-03-04T16:23:56.475932+0000","last_fullsized":"2024-03-04T16:23:56.475932+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:22:52.714182+0000","last_deep_scrub":"316'24","last_deep_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.714182+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.278985+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":722,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839302+0000","last_change":"2024-03-04T16:23:11.346983+0000","last_active":"2024-03-04T16:23:50.839302+0000","last_peered":"2024-03-04T16:23:50.839302+0000","last_clean":"2024-03-04T16:23:50.839302+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:23:50.839302+0000","last_undegraded":"2024-03-04T16:23:50.839302+0000","last_fullsized":"2024-03-04T16:23:50.839302+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:11.346910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:11.346910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.225627+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":1120,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835148+0000","last_change":"2024-03-04T16:22:04.841240+0000","last_active":"2024-03-04T16:23:50.835148+0000","last_peered":"2024-03-04T16:23:50.835148+0000","last_clean":"2024-03-04T16:23:50.835148+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:23:50.835148+0000","last_undegraded":"2024-03-04T16:23:50.835148+0000","last_fullsized":"2024-03-04T16:23:50.835148+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:22:04.841055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.841055+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.079956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":918,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837439+0000","last_change":"2024-03-04T16:23:03.314418+0000","last_active":"2024-03-04T16:23:50.837439+0000","last_peered":"2024-03-04T16:23:50.837439+0000","last_clean":"2024-03-04T16:23:50.837439+0000","last_became_active":"2024-03-04T16:21:45.413361+0000","last_became_peered":"2024-03-04T16:21:45.413361+0000","last_unstale":"2024-03-04T16:23:50.837439+0000","last_undegraded":"2024-03-04T16:23:50.837439+0000","last_fullsized":"2024-03-04T16:23:50.837439+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:23:03.314294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.314294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:26.292076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'30","reported_seq":575,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875847+0000","last_change":"2024-03-04T16:21:35.752373+0000","last_active":"2024-03-04T16:23:53.875847+0000","last_peered":"2024-03-04T16:23:53.875847+0000","last_clean":"2024-03-04T16:23:53.875847+0000","last_became_active":"2024-03-04T16:21:25.452317+0000","last_became_peered":"2024-03-04T16:21:25.452317+0000","last_unstale":"2024-03-04T16:23:53.875847+0000","last_undegraded":"2024-03-04T16:23:53.875847+0000","last_fullsized":"2024-03-04T16:23:53.875847+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'20","last_scrub_stamp":"2024-03-04T16:21:35.752284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:35.752284+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":801,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.849301+0000","last_change":"2024-03-04T16:23:09.663195+0000","last_active":"2024-03-04T16:23:50.849301+0000","last_peered":"2024-03-04T16:23:50.849301+0000","last_clean":"2024-03-04T16:23:50.849301+0000","last_became_active":"2024-03-04T16:20:07.486320+0000","last_became_peered":"2024-03-04T16:20:07.486320+0000","last_unstale":"2024-03-04T16:23:50.849301+0000","last_undegraded":"2024-03-04T16:23:50.849301+0000","last_fullsized":"2024-03-04T16:23:50.849301+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:09.662860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.662860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.759094+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"338'11","reported_seq":1116,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.030699+0000","last_change":"2024-03-04T16:22:57.950248+0000","last_active":"2024-03-04T16:23:54.030699+0000","last_peered":"2024-03-04T16:23:54.030699+0000","last_clean":"2024-03-04T16:23:54.030699+0000","last_became_active":"2024-03-04T16:19:04.780989+0000","last_became_peered":"2024-03-04T16:19:04.780989+0000","last_unstale":"2024-03-04T16:23:54.030699+0000","last_undegraded":"2024-03-04T16:23:54.030699+0000","last_fullsized":"2024-03-04T16:23:54.030699+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:57.950104+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.950104+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:06.619332+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":747,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.836975+0000","last_change":"2024-03-04T16:22:00.982774+0000","last_active":"2024-03-04T16:23:50.836975+0000","last_peered":"2024-03-04T16:23:50.836975+0000","last_clean":"2024-03-04T16:23:50.836975+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:23:50.836975+0000","last_undegraded":"2024-03-04T16:23:50.836975+0000","last_fullsized":"2024-03-04T16:23:50.836975+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:00.982672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:00.982672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"441'21","reported_seq":882,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835550+0000","last_change":"2024-03-04T16:22:56.048086+0000","last_active":"2024-03-04T16:23:50.835550+0000","last_peered":"2024-03-04T16:23:50.835550+0000","last_clean":"2024-03-04T16:23:50.835550+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:23:50.835550+0000","last_undegraded":"2024-03-04T16:23:50.835550+0000","last_fullsized":"2024-03-04T16:23:50.835550+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:22:56.047875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.047875+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.270041+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":257,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.848513+0000","last_change":"2024-03-04T16:21:54.866369+0000","last_active":"2024-03-04T16:23:50.848513+0000","last_peered":"2024-03-04T16:23:50.848513+0000","last_clean":"2024-03-04T16:23:50.848513+0000","last_became_active":"2024-03-04T16:20:07.491512+0000","last_became_peered":"2024-03-04T16:20:07.491512+0000","last_unstale":"2024-03-04T16:23:50.848513+0000","last_undegraded":"2024-03-04T16:23:50.848513+0000","last_fullsized":"2024-03-04T16:23:50.848513+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:54.866103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:54.866103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057374+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":831,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.864289+0000","last_change":"2024-03-04T16:22:41.839730+0000","last_active":"2024-03-04T16:23:53.864289+0000","last_peered":"2024-03-04T16:23:53.864289+0000","last_clean":"2024-03-04T16:23:53.864289+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:23:53.864289+0000","last_undegraded":"2024-03-04T16:23:53.864289+0000","last_fullsized":"2024-03-04T16:23:53.864289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.839621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.839621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.180908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":864,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.836913+0000","last_change":"2024-03-04T16:21:45.415202+0000","last_active":"2024-03-04T16:23:50.836913+0000","last_peered":"2024-03-04T16:23:50.836913+0000","last_clean":"2024-03-04T16:23:50.836913+0000","last_became_active":"2024-03-04T16:21:45.414717+0000","last_became_peered":"2024-03-04T16:21:45.414717+0000","last_unstale":"2024-03-04T16:23:50.836913+0000","last_undegraded":"2024-03-04T16:23:50.836913+0000","last_fullsized":"2024-03-04T16:23:50.836913+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_clean_scrub_stamp":"2024-03-04T16:19:30.702539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284679+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'21","reported_seq":719,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874280+0000","last_change":"2024-03-04T16:21:40.428230+0000","last_active":"2024-03-04T16:23:53.874280+0000","last_peered":"2024-03-04T16:23:53.874280+0000","last_clean":"2024-03-04T16:23:53.874280+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:23:53.874280+0000","last_undegraded":"2024-03-04T16:23:53.874280+0000","last_fullsized":"2024-03-04T16:23:53.874280+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:21:40.428083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.428083+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.574927+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":859,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837695+0000","last_change":"2024-03-04T16:21:45.412628+0000","last_active":"2024-03-04T16:23:50.837695+0000","last_peered":"2024-03-04T16:23:50.837695+0000","last_clean":"2024-03-04T16:23:50.837695+0000","last_became_active":"2024-03-04T16:21:45.411986+0000","last_became_peered":"2024-03-04T16:21:45.411986+0000","last_unstale":"2024-03-04T16:23:50.837695+0000","last_undegraded":"2024-03-04T16:23:50.837695+0000","last_fullsized":"2024-03-04T16:23:50.837695+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:59.610360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:20:59.610360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284708+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":836,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835120+0000","last_change":"2024-03-04T16:20:59.503150+0000","last_active":"2024-03-04T16:23:50.835120+0000","last_peered":"2024-03-04T16:23:50.835120+0000","last_clean":"2024-03-04T16:23:50.835120+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:23:50.835120+0000","last_undegraded":"2024-03-04T16:23:50.835120+0000","last_fullsized":"2024-03-04T16:23:50.835120+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:59.503052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:20:59.503052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078248+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":103,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.477827+0000","last_change":"2024-03-04T16:21:45.414340+0000","last_active":"2024-03-04T16:23:56.477827+0000","last_peered":"2024-03-04T16:23:56.477827+0000","last_clean":"2024-03-04T16:23:56.477827+0000","last_became_active":"2024-03-04T16:21:45.413989+0000","last_became_peered":"2024-03-04T16:21:45.413989+0000","last_unstale":"2024-03-04T16:23:56.477827+0000","last_undegraded":"2024-03-04T16:23:56.477827+0000","last_fullsized":"2024-03-04T16:23:56.477827+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:39.439330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:21:39.439330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'21","reported_seq":865,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.864454+0000","last_change":"2024-03-04T16:22:44.928303+0000","last_active":"2024-03-04T16:23:53.864454+0000","last_peered":"2024-03-04T16:23:53.864454+0000","last_clean":"2024-03-04T16:23:53.864454+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:23:53.864454+0000","last_undegraded":"2024-03-04T16:23:53.864454+0000","last_fullsized":"2024-03-04T16:23:53.864454+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:22:44.928118+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.928118+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048904+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"453'9848","reported_seq":9963,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.127009+0000","last_change":"2024-03-04T16:21:25.449309+0000","last_active":"2024-03-04T16:23:53.127009+0000","last_peered":"2024-03-04T16:23:53.127009+0000","last_clean":"2024-03-04T16:23:53.127009+0000","last_became_active":"2024-03-04T16:21:25.449029+0000","last_became_peered":"2024-03-04T16:21:25.449029+0000","last_unstale":"2024-03-04T16:23:53.127009+0000","last_undegraded":"2024-03-04T16:23:53.127009+0000","last_fullsized":"2024-03-04T16:23:53.127009+0000","mapping_epoch":401,"log_start":"437'4900","ondisk_log_start":"437'4900","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228720640,"num_objects":3490,"num_object_clones":0,"num_object_copies":10470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9848,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":859,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834314+0000","last_change":"2024-03-04T16:23:04.261493+0000","last_active":"2024-03-04T16:23:50.834314+0000","last_peered":"2024-03-04T16:23:50.834314+0000","last_clean":"2024-03-04T16:23:50.834314+0000","last_became_active":"2024-03-04T16:21:45.424279+0000","last_became_peered":"2024-03-04T16:21:45.424279+0000","last_unstale":"2024-03-04T16:23:50.834314+0000","last_undegraded":"2024-03-04T16:23:50.834314+0000","last_fullsized":"2024-03-04T16:23:50.834314+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.261358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.261358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.709711+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":902,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837738+0000","last_change":"2024-03-04T16:21:07.575776+0000","last_active":"2024-03-04T16:23:50.837738+0000","last_peered":"2024-03-04T16:23:50.837738+0000","last_clean":"2024-03-04T16:23:50.837738+0000","last_became_active":"2024-03-04T16:20:07.493115+0000","last_became_peered":"2024-03-04T16:20:07.493115+0000","last_unstale":"2024-03-04T16:23:50.837738+0000","last_undegraded":"2024-03-04T16:23:50.837738+0000","last_fullsized":"2024-03-04T16:23:50.837738+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:21:07.575695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:07.575695+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":976,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.833816+0000","last_change":"2024-03-04T16:23:00.064299+0000","last_active":"2024-03-04T16:23:50.833816+0000","last_peered":"2024-03-04T16:23:50.833816+0000","last_clean":"2024-03-04T16:23:50.833816+0000","last_became_active":"2024-03-04T16:21:45.422336+0000","last_became_peered":"2024-03-04T16:21:45.422336+0000","last_unstale":"2024-03-04T16:23:50.833816+0000","last_undegraded":"2024-03-04T16:23:50.833816+0000","last_fullsized":"2024-03-04T16:23:50.833816+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:23:00.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.064148+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.044607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"412'11","reported_seq":1146,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837153+0000","last_change":"2024-03-04T16:23:07.273194+0000","last_active":"2024-03-04T16:23:50.837153+0000","last_peered":"2024-03-04T16:23:50.837153+0000","last_clean":"2024-03-04T16:23:50.837153+0000","last_became_active":"2024-03-04T16:21:45.441048+0000","last_became_peered":"2024-03-04T16:21:45.441048+0000","last_unstale":"2024-03-04T16:23:50.837153+0000","last_undegraded":"2024-03-04T16:23:50.837153+0000","last_fullsized":"2024-03-04T16:23:50.837153+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'11","last_scrub_stamp":"2024-03-04T16:23:07.273057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.273057+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:35.024574+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"454'9793","reported_seq":9982,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.316361+0000","last_change":"2024-03-04T16:23:06.658833+0000","last_active":"2024-03-04T16:23:54.316361+0000","last_peered":"2024-03-04T16:23:54.316361+0000","last_clean":"2024-03-04T16:23:54.316361+0000","last_became_active":"2024-03-04T16:21:25.448720+0000","last_became_peered":"2024-03-04T16:21:25.448720+0000","last_unstale":"2024-03-04T16:23:54.316361+0000","last_undegraded":"2024-03-04T16:23:54.316361+0000","last_fullsized":"2024-03-04T16:23:54.316361+0000","mapping_epoch":401,"log_start":"439'5100","ondisk_log_start":"439'5100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6580","last_scrub_stamp":"2024-03-04T16:23:06.658440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.658440+0000","objects_scrubbed":6489,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.080402+0000","scrub_duration":3051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222363648,"num_objects":3393,"num_object_clones":0,"num_object_copies":10179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9793,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":828,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.867203+0000","last_change":"2024-03-04T16:21:41.947473+0000","last_active":"2024-03-04T16:23:53.867203+0000","last_peered":"2024-03-04T16:23:53.867203+0000","last_clean":"2024-03-04T16:23:53.867203+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:23:53.867203+0000","last_undegraded":"2024-03-04T16:23:53.867203+0000","last_fullsized":"2024-03-04T16:23:53.867203+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:41.947350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.947350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048786+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":868,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839539+0000","last_change":"2024-03-04T16:22:50.302556+0000","last_active":"2024-03-04T16:23:50.839539+0000","last_peered":"2024-03-04T16:23:50.839539+0000","last_clean":"2024-03-04T16:23:50.839539+0000","last_became_active":"2024-03-04T16:20:07.497627+0000","last_became_peered":"2024-03-04T16:20:07.497627+0000","last_unstale":"2024-03-04T16:23:50.839539+0000","last_undegraded":"2024-03-04T16:23:50.839539+0000","last_fullsized":"2024-03-04T16:23:50.839539+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.302482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.302482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.673861+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":663,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875238+0000","last_change":"2024-03-04T16:22:52.266342+0000","last_active":"2024-03-04T16:23:53.875238+0000","last_peered":"2024-03-04T16:23:53.875238+0000","last_clean":"2024-03-04T16:23:53.875238+0000","last_became_active":"2024-03-04T16:21:45.443501+0000","last_became_peered":"2024-03-04T16:21:45.443501+0000","last_unstale":"2024-03-04T16:23:53.875238+0000","last_undegraded":"2024-03-04T16:23:53.875238+0000","last_fullsized":"2024-03-04T16:23:53.875238+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:22:52.266216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.266216+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.312618+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":251,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.847671+0000","last_change":"2024-03-04T16:22:02.701368+0000","last_active":"2024-03-04T16:23:50.847671+0000","last_peered":"2024-03-04T16:23:50.847671+0000","last_clean":"2024-03-04T16:23:50.847671+0000","last_became_active":"2024-03-04T16:19:04.768239+0000","last_became_peered":"2024-03-04T16:19:04.768239+0000","last_unstale":"2024-03-04T16:23:50.847671+0000","last_undegraded":"2024-03-04T16:23:50.847671+0000","last_fullsized":"2024-03-04T16:23:50.847671+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:02.701238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:02.701238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"453'9897","reported_seq":12078,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:52.791705+0000","last_change":"2024-03-04T16:21:59.472159+0000","last_active":"2024-03-04T16:23:52.791705+0000","last_peered":"2024-03-04T16:23:52.791705+0000","last_clean":"2024-03-04T16:23:52.791705+0000","last_became_active":"2024-03-04T16:21:59.469208+0000","last_became_peered":"2024-03-04T16:21:59.469208+0000","last_unstale":"2024-03-04T16:23:52.791705+0000","last_undegraded":"2024-03-04T16:23:52.791705+0000","last_fullsized":"2024-03-04T16:23:52.791705+0000","mapping_epoch":421,"log_start":"439'5200","ondisk_log_start":"439'5200","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4697,"log_dups_size":0,"ondisk_log_size":4697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":235208704,"num_objects":3589,"num_object_clones":0,"num_object_copies":10767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9897,"num_write_kb":431552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":67305472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":860,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837388+0000","last_change":"2024-03-04T16:21:44.902269+0000","last_active":"2024-03-04T16:23:50.837388+0000","last_peered":"2024-03-04T16:23:50.837388+0000","last_clean":"2024-03-04T16:23:50.837388+0000","last_became_active":"2024-03-04T16:19:04.749223+0000","last_became_peered":"2024-03-04T16:19:04.749223+0000","last_unstale":"2024-03-04T16:23:50.837388+0000","last_undegraded":"2024-03-04T16:23:50.837388+0000","last_fullsized":"2024-03-04T16:23:50.837388+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.902188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.902188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":251,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.848195+0000","last_change":"2024-03-04T16:22:04.717449+0000","last_active":"2024-03-04T16:23:50.848195+0000","last_peered":"2024-03-04T16:23:50.848195+0000","last_clean":"2024-03-04T16:23:50.848195+0000","last_became_active":"2024-03-04T16:19:04.780445+0000","last_became_peered":"2024-03-04T16:19:04.780445+0000","last_unstale":"2024-03-04T16:23:50.848195+0000","last_undegraded":"2024-03-04T16:23:50.848195+0000","last_fullsized":"2024-03-04T16:23:50.848195+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:04.717368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.717368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":866,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.838135+0000","last_change":"2024-03-04T16:21:59.968201+0000","last_active":"2024-03-04T16:23:50.838135+0000","last_peered":"2024-03-04T16:23:50.838135+0000","last_clean":"2024-03-04T16:23:50.838135+0000","last_became_active":"2024-03-04T16:21:45.411738+0000","last_became_peered":"2024-03-04T16:21:45.411738+0000","last_unstale":"2024-03-04T16:23:50.838135+0000","last_undegraded":"2024-03-04T16:23:50.838135+0000","last_fullsized":"2024-03-04T16:23:50.838135+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:59.968123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:21:59.968123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'52","reported_seq":182,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476536+0000","last_change":"2024-03-04T16:22:51.690727+0000","last_active":"2024-03-04T16:23:56.476536+0000","last_peered":"2024-03-04T16:23:56.476536+0000","last_clean":"2024-03-04T16:23:56.476536+0000","last_became_active":"2024-03-04T16:22:24.826414+0000","last_became_peered":"2024-03-04T16:22:24.826414+0000","last_unstale":"2024-03-04T16:23:56.476536+0000","last_undegraded":"2024-03-04T16:23:56.476536+0000","last_fullsized":"2024-03-04T16:23:56.476536+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:22:51.690576+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.690576+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:05.795083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"453'9900","reported_seq":12068,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:52.817193+0000","last_change":"2024-03-04T16:21:52.427898+0000","last_active":"2024-03-04T16:23:52.817193+0000","last_peered":"2024-03-04T16:23:52.817193+0000","last_clean":"2024-03-04T16:23:52.817193+0000","last_became_active":"2024-03-04T16:21:52.426535+0000","last_became_peered":"2024-03-04T16:21:52.426535+0000","last_unstale":"2024-03-04T16:23:52.817193+0000","last_undegraded":"2024-03-04T16:23:52.817193+0000","last_fullsized":"2024-03-04T16:23:52.817193+0000","mapping_epoch":414,"log_start":"439'5200","ondisk_log_start":"439'5200","created":401,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237109248,"num_objects":3618,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":66977792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":735,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.836754+0000","last_change":"2024-03-04T16:22:06.237495+0000","last_active":"2024-03-04T16:23:50.836754+0000","last_peered":"2024-03-04T16:23:50.836754+0000","last_clean":"2024-03-04T16:23:50.836754+0000","last_became_active":"2024-03-04T16:19:04.779570+0000","last_became_peered":"2024-03-04T16:19:04.779570+0000","last_unstale":"2024-03-04T16:23:50.836754+0000","last_undegraded":"2024-03-04T16:23:50.836754+0000","last_fullsized":"2024-03-04T16:23:50.836754+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.237403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.237403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286284+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":265,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.850041+0000","last_change":"2024-03-04T16:22:54.512261+0000","last_active":"2024-03-04T16:23:50.850041+0000","last_peered":"2024-03-04T16:23:50.850041+0000","last_clean":"2024-03-04T16:23:50.850041+0000","last_became_active":"2024-03-04T16:20:07.487240+0000","last_became_peered":"2024-03-04T16:20:07.487240+0000","last_unstale":"2024-03-04T16:23:50.850041+0000","last_undegraded":"2024-03-04T16:23:50.850041+0000","last_fullsized":"2024-03-04T16:23:50.850041+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.512143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.512143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:01.584413+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":865,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834907+0000","last_change":"2024-03-04T16:23:03.094410+0000","last_active":"2024-03-04T16:23:50.834907+0000","last_peered":"2024-03-04T16:23:50.834907+0000","last_clean":"2024-03-04T16:23:50.834907+0000","last_became_active":"2024-03-04T16:21:45.422860+0000","last_became_peered":"2024-03-04T16:21:45.422860+0000","last_unstale":"2024-03-04T16:23:50.834907+0000","last_undegraded":"2024-03-04T16:23:50.834907+0000","last_fullsized":"2024-03-04T16:23:50.834907+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.094190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.094190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.574130+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"441'25","reported_seq":760,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874182+0000","last_change":"2024-03-04T16:22:56.668460+0000","last_active":"2024-03-04T16:23:53.874182+0000","last_peered":"2024-03-04T16:23:53.874182+0000","last_clean":"2024-03-04T16:23:53.874182+0000","last_became_active":"2024-03-04T16:21:10.564447+0000","last_became_peered":"2024-03-04T16:21:10.564447+0000","last_unstale":"2024-03-04T16:23:53.874182+0000","last_undegraded":"2024-03-04T16:23:53.874182+0000","last_fullsized":"2024-03-04T16:23:53.874182+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:22:56.668180+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.668180+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.974798+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"454'10062","reported_seq":10239,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.387726+0000","last_change":"2024-03-04T16:22:42.339861+0000","last_active":"2024-03-04T16:23:54.387726+0000","last_peered":"2024-03-04T16:23:54.387726+0000","last_clean":"2024-03-04T16:23:54.387726+0000","last_became_active":"2024-03-04T16:21:25.449702+0000","last_became_peered":"2024-03-04T16:21:25.449702+0000","last_unstale":"2024-03-04T16:23:54.387726+0000","last_undegraded":"2024-03-04T16:23:54.387726+0000","last_fullsized":"2024-03-04T16:23:54.387726+0000","mapping_epoch":401,"log_start":"439'5500","ondisk_log_start":"439'5500","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5208","last_scrub_stamp":"2024-03-04T16:22:42.339680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.339680+0000","objects_scrubbed":5130,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.727729+0000","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219283456,"num_objects":3346,"num_object_clones":0,"num_object_copies":10038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10062,"num_write_kb":429056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":840,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.865525+0000","last_change":"2024-03-04T16:22:52.913981+0000","last_active":"2024-03-04T16:23:53.865525+0000","last_peered":"2024-03-04T16:23:53.865525+0000","last_clean":"2024-03-04T16:23:53.865525+0000","last_became_active":"2024-03-04T16:21:45.423440+0000","last_became_peered":"2024-03-04T16:21:45.423440+0000","last_unstale":"2024-03-04T16:23:53.865525+0000","last_undegraded":"2024-03-04T16:23:53.865525+0000","last_fullsized":"2024-03-04T16:23:53.865525+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:52.913842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.913842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.335302+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834151+0000","last_change":"2024-03-04T16:21:45.446577+0000","last_active":"2024-03-04T16:23:50.834151+0000","last_peered":"2024-03-04T16:23:50.834151+0000","last_clean":"2024-03-04T16:23:50.834151+0000","last_became_active":"2024-03-04T16:21:45.425232+0000","last_became_peered":"2024-03-04T16:21:45.425232+0000","last_unstale":"2024-03-04T16:23:50.834151+0000","last_undegraded":"2024-03-04T16:23:50.834151+0000","last_fullsized":"2024-03-04T16:23:50.834151+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:20:52.438469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:20:52.438469+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078073+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":112,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.475634+0000","last_change":"2024-03-04T16:22:45.604898+0000","last_active":"2024-03-04T16:23:56.475634+0000","last_peered":"2024-03-04T16:23:56.475634+0000","last_clean":"2024-03-04T16:23:56.475634+0000","last_became_active":"2024-03-04T16:21:45.437679+0000","last_became_peered":"2024-03-04T16:21:45.437679+0000","last_unstale":"2024-03-04T16:23:56.475634+0000","last_undegraded":"2024-03-04T16:23:56.475634+0000","last_fullsized":"2024-03-04T16:23:56.475634+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.604769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.604769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.372412+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"441'55","reported_seq":1276,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876334+0000","last_change":"2024-03-04T16:22:04.783731+0000","last_active":"2024-03-04T16:23:53.876334+0000","last_peered":"2024-03-04T16:23:53.876334+0000","last_clean":"2024-03-04T16:23:53.876334+0000","last_became_active":"2024-03-04T16:19:04.769086+0000","last_became_peered":"2024-03-04T16:19:04.769086+0000","last_unstale":"2024-03-04T16:23:53.876334+0000","last_undegraded":"2024-03-04T16:23:53.876334+0000","last_fullsized":"2024-03-04T16:23:53.876334+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'43","last_scrub_stamp":"2024-03-04T16:22:04.783619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.783619+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":704,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.871035+0000","last_change":"2024-03-04T16:21:45.434819+0000","last_active":"2024-03-04T16:23:53.871035+0000","last_peered":"2024-03-04T16:23:53.871035+0000","last_clean":"2024-03-04T16:23:53.871035+0000","last_became_active":"2024-03-04T16:21:45.434502+0000","last_became_peered":"2024-03-04T16:21:45.434502+0000","last_unstale":"2024-03-04T16:23:53.871035+0000","last_undegraded":"2024-03-04T16:23:53.871035+0000","last_fullsized":"2024-03-04T16:23:53.871035+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.410657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.410657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.574967+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"455'10075","reported_seq":12316,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:57.597331+0000","last_change":"2024-03-04T16:22:04.820773+0000","last_active":"2024-03-04T16:23:57.597331+0000","last_peered":"2024-03-04T16:23:57.597331+0000","last_clean":"2024-03-04T16:23:57.597331+0000","last_became_active":"2024-03-04T16:22:04.786993+0000","last_became_peered":"2024-03-04T16:22:04.786993+0000","last_unstale":"2024-03-04T16:23:57.597331+0000","last_undegraded":"2024-03-04T16:23:57.597331+0000","last_fullsized":"2024-03-04T16:23:57.597331+0000","mapping_epoch":424,"log_start":"439'5200","ondisk_log_start":"439'5200","created":401,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4875,"log_dups_size":0,"ondisk_log_size":4875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210042880,"num_objects":3205,"num_object_clones":0,"num_object_copies":9615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10075,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":69074944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":861,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.838266+0000","last_change":"2024-03-04T16:21:45.415808+0000","last_active":"2024-03-04T16:23:50.838266+0000","last_peered":"2024-03-04T16:23:50.838266+0000","last_clean":"2024-03-04T16:23:50.838266+0000","last_became_active":"2024-03-04T16:21:45.415473+0000","last_became_peered":"2024-03-04T16:21:45.415473+0000","last_unstale":"2024-03-04T16:23:50.838266+0000","last_undegraded":"2024-03-04T16:23:50.838266+0000","last_fullsized":"2024-03-04T16:23:50.838266+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:40.973478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.973478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284822+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":881,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875375+0000","last_change":"2024-03-04T16:22:51.222795+0000","last_active":"2024-03-04T16:23:53.875375+0000","last_peered":"2024-03-04T16:23:53.875375+0000","last_clean":"2024-03-04T16:23:53.875375+0000","last_became_active":"2024-03-04T16:21:45.437084+0000","last_became_peered":"2024-03-04T16:21:45.437084+0000","last_unstale":"2024-03-04T16:23:53.875375+0000","last_undegraded":"2024-03-04T16:23:53.875375+0000","last_fullsized":"2024-03-04T16:23:53.875375+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:51.222677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.222677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.054940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.852436+0000","last_change":"2024-03-04T16:22:09.686124+0000","last_active":"2024-03-04T16:23:50.852436+0000","last_peered":"2024-03-04T16:23:50.852436+0000","last_clean":"2024-03-04T16:23:50.852436+0000","last_became_active":"2024-03-04T16:19:04.780845+0000","last_became_peered":"2024-03-04T16:19:04.780845+0000","last_unstale":"2024-03-04T16:23:50.852436+0000","last_undegraded":"2024-03-04T16:23:50.852436+0000","last_fullsized":"2024-03-04T16:23:50.852436+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.685989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.685989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057495+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":506,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.881701+0000","last_change":"2024-03-04T16:23:08.368485+0000","last_active":"2024-03-04T16:23:53.881701+0000","last_peered":"2024-03-04T16:23:53.881701+0000","last_clean":"2024-03-04T16:23:53.881701+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:23:53.881701+0000","last_undegraded":"2024-03-04T16:23:53.881701+0000","last_fullsized":"2024-03-04T16:23:53.881701+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-04T16:23:08.368095+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.368095+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.910733+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":259,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.848970+0000","last_change":"2024-03-04T16:22:43.622761+0000","last_active":"2024-03-04T16:23:50.848970+0000","last_peered":"2024-03-04T16:23:50.848970+0000","last_clean":"2024-03-04T16:23:50.848970+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:23:50.848970+0000","last_undegraded":"2024-03-04T16:23:50.848970+0000","last_fullsized":"2024-03-04T16:23:50.848970+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.622383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.622383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.378061+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"455'9987","reported_seq":12115,"reported_epoch":455,"state":"active+clean+scrubbing","last_fresh":"2024-03-04T16:23:57.633126+0000","last_change":"2024-03-04T16:23:57.035584+0000","last_active":"2024-03-04T16:23:57.633126+0000","last_peered":"2024-03-04T16:23:57.633126+0000","last_clean":"2024-03-04T16:23:57.633126+0000","last_became_active":"2024-03-04T16:21:58.469058+0000","last_became_peered":"2024-03-04T16:21:58.469058+0000","last_unstale":"2024-03-04T16:23:57.633126+0000","last_undegraded":"2024-03-04T16:23:57.633126+0000","last_fullsized":"2024-03-04T16:23:57.633126+0000","mapping_epoch":420,"log_start":"439'5100","ondisk_log_start":"439'5100","created":401,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":2599,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9987,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":64487424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":847,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835819+0000","last_change":"2024-03-04T16:22:50.355727+0000","last_active":"2024-03-04T16:23:50.835819+0000","last_peered":"2024-03-04T16:23:50.835819+0000","last_clean":"2024-03-04T16:23:50.835819+0000","last_became_active":"2024-03-04T16:20:07.500640+0000","last_became_peered":"2024-03-04T16:20:07.500640+0000","last_unstale":"2024-03-04T16:23:50.835819+0000","last_undegraded":"2024-03-04T16:23:50.835819+0000","last_fullsized":"2024-03-04T16:23:50.835819+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.355598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.355598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.959365+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1177,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837568+0000","last_change":"2024-03-04T16:22:48.307804+0000","last_active":"2024-03-04T16:23:50.837568+0000","last_peered":"2024-03-04T16:23:50.837568+0000","last_clean":"2024-03-04T16:23:50.837568+0000","last_became_active":"2024-03-04T16:21:48.434375+0000","last_became_peered":"2024-03-04T16:21:48.434375+0000","last_unstale":"2024-03-04T16:23:50.837568+0000","last_undegraded":"2024-03-04T16:23:50.837568+0000","last_fullsized":"2024-03-04T16:23:50.837568+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:22:48.307675+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.307675+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.653556+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":777,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834102+0000","last_change":"2024-03-04T16:21:45.425510+0000","last_active":"2024-03-04T16:23:50.834102+0000","last_peered":"2024-03-04T16:23:50.834102+0000","last_clean":"2024-03-04T16:23:50.834102+0000","last_became_active":"2024-03-04T16:21:45.424708+0000","last_became_peered":"2024-03-04T16:21:45.424708+0000","last_unstale":"2024-03-04T16:23:50.834102+0000","last_undegraded":"2024-03-04T16:23:50.834102+0000","last_fullsized":"2024-03-04T16:23:50.834102+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:03.041001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:03.041001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078105+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"441'51","reported_seq":174,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476115+0000","last_change":"2024-03-04T16:23:51.050483+0000","last_active":"2024-03-04T16:23:56.476115+0000","last_peered":"2024-03-04T16:23:56.476115+0000","last_clean":"2024-03-04T16:23:56.476115+0000","last_became_active":"2024-03-04T16:21:45.411357+0000","last_became_peered":"2024-03-04T16:21:45.411357+0000","last_unstale":"2024-03-04T16:23:56.476115+0000","last_undegraded":"2024-03-04T16:23:56.476115+0000","last_fullsized":"2024-03-04T16:23:56.476115+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'51","last_scrub_stamp":"2024-03-04T16:23:51.050351+0000","last_deep_scrub":"391'38","last_deep_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_clean_scrub_stamp":"2024-03-04T16:23:51.050351+0000","objects_scrubbed":8,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:06.142506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":867,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.839812+0000","last_change":"2024-03-04T16:22:04.052615+0000","last_active":"2024-03-04T16:23:50.839812+0000","last_peered":"2024-03-04T16:23:50.839812+0000","last_clean":"2024-03-04T16:23:50.839812+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:23:50.839812+0000","last_undegraded":"2024-03-04T16:23:50.839812+0000","last_fullsized":"2024-03-04T16:23:50.839812+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:04.052531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.052531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"454'9885","reported_seq":12118,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.449330+0000","last_change":"2024-03-04T16:21:55.410964+0000","last_active":"2024-03-04T16:23:54.449330+0000","last_peered":"2024-03-04T16:23:54.449330+0000","last_clean":"2024-03-04T16:23:54.449330+0000","last_became_active":"2024-03-04T16:21:55.408063+0000","last_became_peered":"2024-03-04T16:21:55.408063+0000","last_unstale":"2024-03-04T16:23:54.449330+0000","last_undegraded":"2024-03-04T16:23:54.449330+0000","last_fullsized":"2024-03-04T16:23:54.449330+0000","mapping_epoch":417,"log_start":"439'5400","ondisk_log_start":"439'5400","created":401,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4485,"log_dups_size":0,"ondisk_log_size":4485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220659712,"num_objects":3367,"num_object_clones":0,"num_object_copies":10101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9885,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":69009408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":853,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875788+0000","last_change":"2024-03-04T16:23:03.390202+0000","last_active":"2024-03-04T16:23:53.875788+0000","last_peered":"2024-03-04T16:23:53.875788+0000","last_clean":"2024-03-04T16:23:53.875788+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:23:53.875788+0000","last_undegraded":"2024-03-04T16:23:53.875788+0000","last_fullsized":"2024-03-04T16:23:53.875788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.389804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.389804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.921181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":706,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870843+0000","last_change":"2024-03-04T16:21:38.419849+0000","last_active":"2024-03-04T16:23:53.870843+0000","last_peered":"2024-03-04T16:23:53.870843+0000","last_clean":"2024-03-04T16:23:53.870843+0000","last_became_active":"2024-03-04T16:20:07.498576+0000","last_became_peered":"2024-03-04T16:20:07.498576+0000","last_unstale":"2024-03-04T16:23:53.870843+0000","last_undegraded":"2024-03-04T16:23:53.870843+0000","last_fullsized":"2024-03-04T16:23:53.870843+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:38.419741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:38.419741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.574994+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":2137,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.892870+0000","last_change":"2024-03-04T16:21:41.881246+0000","last_active":"2024-03-04T16:23:50.892870+0000","last_peered":"2024-03-04T16:23:50.892870+0000","last_clean":"2024-03-04T16:23:50.892870+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:23:50.892870+0000","last_undegraded":"2024-03-04T16:23:50.892870+0000","last_fullsized":"2024-03-04T16:23:50.892870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:21:41.881125+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.881125+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.286200+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1261,"num_read_kb":1256,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":802,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.873965+0000","last_change":"2024-03-04T16:22:24.868227+0000","last_active":"2024-03-04T16:23:53.873965+0000","last_peered":"2024-03-04T16:23:53.873965+0000","last_clean":"2024-03-04T16:23:53.873965+0000","last_became_active":"2024-03-04T16:22:24.838293+0000","last_became_peered":"2024-03-04T16:22:24.838293+0000","last_unstale":"2024-03-04T16:23:53.873965+0000","last_undegraded":"2024-03-04T16:23:53.873965+0000","last_fullsized":"2024-03-04T16:23:53.873965+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'24","last_scrub_stamp":"2024-03-04T16:22:09.593105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.593105+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:55.575024+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":853,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.836509+0000","last_change":"2024-03-04T16:22:45.128630+0000","last_active":"2024-03-04T16:23:50.836509+0000","last_peered":"2024-03-04T16:23:50.836509+0000","last_clean":"2024-03-04T16:23:50.836509+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:23:50.836509+0000","last_undegraded":"2024-03-04T16:23:50.836509+0000","last_fullsized":"2024-03-04T16:23:50.836509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.128548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.128548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.080014+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"454'9896","reported_seq":10097,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.431822+0000","last_change":"2024-03-04T16:23:53.898834+0000","last_active":"2024-03-04T16:23:54.431822+0000","last_peered":"2024-03-04T16:23:54.431822+0000","last_clean":"2024-03-04T16:23:54.431822+0000","last_became_active":"2024-03-04T16:21:25.448277+0000","last_became_peered":"2024-03-04T16:21:25.448277+0000","last_unstale":"2024-03-04T16:23:54.431822+0000","last_undegraded":"2024-03-04T16:23:54.431822+0000","last_fullsized":"2024-03-04T16:23:54.431822+0000","mapping_epoch":401,"log_start":"441'5600","ondisk_log_start":"441'5600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'9876","last_scrub_stamp":"2024-03-04T16:23:53.898498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:53.898498+0000","objects_scrubbed":3447,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:23.002547+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221904896,"num_objects":3386,"num_object_clones":0,"num_object_copies":10158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9896,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":869,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875438+0000","last_change":"2024-03-04T16:22:02.934144+0000","last_active":"2024-03-04T16:23:53.875438+0000","last_peered":"2024-03-04T16:23:53.875438+0000","last_clean":"2024-03-04T16:23:53.875438+0000","last_became_active":"2024-03-04T16:21:45.442233+0000","last_became_peered":"2024-03-04T16:21:45.442233+0000","last_unstale":"2024-03-04T16:23:53.875438+0000","last_undegraded":"2024-03-04T16:23:53.875438+0000","last_fullsized":"2024-03-04T16:23:53.875438+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:02.934010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_clean_scrub_stamp":"2024-03-04T16:22:02.934010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":861,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876162+0000","last_change":"2024-03-04T16:22:48.112648+0000","last_active":"2024-03-04T16:23:53.876162+0000","last_peered":"2024-03-04T16:23:53.876162+0000","last_clean":"2024-03-04T16:23:53.876162+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:23:53.876162+0000","last_undegraded":"2024-03-04T16:23:53.876162+0000","last_fullsized":"2024-03-04T16:23:53.876162+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.112512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.112512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.233790+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'47","reported_seq":844,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834366+0000","last_change":"2024-03-04T16:22:49.132475+0000","last_active":"2024-03-04T16:23:50.834366+0000","last_peered":"2024-03-04T16:23:50.834366+0000","last_clean":"2024-03-04T16:23:50.834366+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:23:50.834366+0000","last_undegraded":"2024-03-04T16:23:50.834366+0000","last_fullsized":"2024-03-04T16:23:50.834366+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:22:49.132304+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:22:49.132304+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.744012+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":112,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476268+0000","last_change":"2024-03-04T16:22:08.771819+0000","last_active":"2024-03-04T16:23:56.476268+0000","last_peered":"2024-03-04T16:23:56.476268+0000","last_clean":"2024-03-04T16:23:56.476268+0000","last_became_active":"2024-03-04T16:21:45.412152+0000","last_became_peered":"2024-03-04T16:21:45.412152+0000","last_unstale":"2024-03-04T16:23:56.476268+0000","last_undegraded":"2024-03-04T16:23:56.476268+0000","last_fullsized":"2024-03-04T16:23:56.476268+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_clean_scrub_stamp":"2024-03-04T16:22:08.770293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"453'9828","reported_seq":12082,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.068405+0000","last_change":"2024-03-04T16:22:54.725063+0000","last_active":"2024-03-04T16:23:53.068405+0000","last_peered":"2024-03-04T16:23:53.068405+0000","last_clean":"2024-03-04T16:23:53.068405+0000","last_became_active":"2024-03-04T16:21:54.483830+0000","last_became_peered":"2024-03-04T16:21:54.483830+0000","last_unstale":"2024-03-04T16:23:53.068405+0000","last_undegraded":"2024-03-04T16:23:53.068405+0000","last_fullsized":"2024-03-04T16:23:53.068405+0000","mapping_epoch":416,"log_start":"434'4200","ondisk_log_start":"434'4200","created":401,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'5897","last_scrub_stamp":"2024-03-04T16:22:54.724570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.724570+0000","objects_scrubbed":5842,"log_size":5628,"log_dups_size":0,"ondisk_log_size":5628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.844871+0000","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226557976,"num_objects":3458,"num_object_clones":0,"num_object_copies":10374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9828,"num_write_kb":425089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":865,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870597+0000","last_change":"2024-03-04T16:22:47.720482+0000","last_active":"2024-03-04T16:23:53.870597+0000","last_peered":"2024-03-04T16:23:53.870597+0000","last_clean":"2024-03-04T16:23:53.870597+0000","last_became_active":"2024-03-04T16:19:04.834912+0000","last_became_peered":"2024-03-04T16:19:04.834912+0000","last_unstale":"2024-03-04T16:23:53.870597+0000","last_undegraded":"2024-03-04T16:23:53.870597+0000","last_fullsized":"2024-03-04T16:23:53.870597+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.720278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.720278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.926758+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":864,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835237+0000","last_change":"2024-03-04T16:20:51.445972+0000","last_active":"2024-03-04T16:23:50.835237+0000","last_peered":"2024-03-04T16:23:50.835237+0000","last_clean":"2024-03-04T16:23:50.835237+0000","last_became_active":"2024-03-04T16:19:04.762915+0000","last_became_peered":"2024-03-04T16:19:04.762915+0000","last_unstale":"2024-03-04T16:23:50.835237+0000","last_undegraded":"2024-03-04T16:23:50.835237+0000","last_fullsized":"2024-03-04T16:23:50.835237+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_clean_scrub_stamp":"2024-03-04T16:20:51.445890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:52.078191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":921,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.875830+0000","last_change":"2024-03-04T16:22:00.856182+0000","last_active":"2024-03-04T16:23:53.875830+0000","last_peered":"2024-03-04T16:23:53.875830+0000","last_clean":"2024-03-04T16:23:53.875830+0000","last_became_active":"2024-03-04T16:20:07.504270+0000","last_became_peered":"2024-03-04T16:20:07.504270+0000","last_unstale":"2024-03-04T16:23:53.875830+0000","last_undegraded":"2024-03-04T16:23:53.875830+0000","last_fullsized":"2024-03-04T16:23:53.875830+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'25","last_scrub_stamp":"2024-03-04T16:22:00.856053+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:22:00.856053+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"412'12","reported_seq":142,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.475838+0000","last_change":"2024-03-04T16:23:09.792730+0000","last_active":"2024-03-04T16:23:56.475838+0000","last_peered":"2024-03-04T16:23:56.475838+0000","last_clean":"2024-03-04T16:23:56.475838+0000","last_became_active":"2024-03-04T16:21:45.414139+0000","last_became_peered":"2024-03-04T16:21:45.414139+0000","last_unstale":"2024-03-04T16:23:56.475838+0000","last_undegraded":"2024-03-04T16:23:56.475838+0000","last_fullsized":"2024-03-04T16:23:56.475838+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12","last_scrub_stamp":"2024-03-04T16:23:09.792648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.792648+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:09.926315+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"454'10039","reported_seq":10230,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.392823+0000","last_change":"2024-03-04T16:23:12.469661+0000","last_active":"2024-03-04T16:23:54.392823+0000","last_peered":"2024-03-04T16:23:54.392823+0000","last_clean":"2024-03-04T16:23:54.392823+0000","last_became_active":"2024-03-04T16:21:25.451006+0000","last_became_peered":"2024-03-04T16:21:25.451006+0000","last_unstale":"2024-03-04T16:23:54.392823+0000","last_undegraded":"2024-03-04T16:23:54.392823+0000","last_fullsized":"2024-03-04T16:23:54.392823+0000","mapping_epoch":401,"log_start":"441'5800","ondisk_log_start":"441'5800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7132","last_scrub_stamp":"2024-03-04T16:23:12.469418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.469418+0000","objects_scrubbed":6443,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.303121+0000","scrub_duration":3140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224854016,"num_objects":3431,"num_object_clones":0,"num_object_copies":10293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10039,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":875,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837733+0000","last_change":"2024-03-04T16:23:10.418223+0000","last_active":"2024-03-04T16:23:50.837733+0000","last_peered":"2024-03-04T16:23:50.837733+0000","last_clean":"2024-03-04T16:23:50.837733+0000","last_became_active":"2024-03-04T16:21:45.413068+0000","last_became_peered":"2024-03-04T16:21:45.413068+0000","last_unstale":"2024-03-04T16:23:50.837733+0000","last_undegraded":"2024-03-04T16:23:50.837733+0000","last_fullsized":"2024-03-04T16:23:50.837733+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:10.418117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.418117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.395028+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":714,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.874168+0000","last_change":"2024-03-04T16:22:53.683732+0000","last_active":"2024-03-04T16:23:53.874168+0000","last_peered":"2024-03-04T16:23:53.874168+0000","last_clean":"2024-03-04T16:23:53.874168+0000","last_became_active":"2024-03-04T16:20:07.494182+0000","last_became_peered":"2024-03-04T16:20:07.494182+0000","last_unstale":"2024-03-04T16:23:53.874168+0000","last_undegraded":"2024-03-04T16:23:53.874168+0000","last_fullsized":"2024-03-04T16:23:53.874168+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:53.683136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:53.683136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:00.670431+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'49","reported_seq":712,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.833833+0000","last_change":"2024-03-04T16:23:07.122889+0000","last_active":"2024-03-04T16:23:50.833833+0000","last_peered":"2024-03-04T16:23:50.833833+0000","last_clean":"2024-03-04T16:23:50.833833+0000","last_became_active":"2024-03-04T16:21:45.423250+0000","last_became_peered":"2024-03-04T16:21:45.423250+0000","last_unstale":"2024-03-04T16:23:50.833833+0000","last_undegraded":"2024-03-04T16:23:50.833833+0000","last_fullsized":"2024-03-04T16:23:50.833833+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'49","last_scrub_stamp":"2024-03-04T16:23:07.122781+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.122781+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.026776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"366'9","reported_seq":1106,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.866315+0000","last_change":"2024-03-04T16:22:08.721644+0000","last_active":"2024-03-04T16:23:53.866315+0000","last_peered":"2024-03-04T16:23:53.866315+0000","last_clean":"2024-03-04T16:23:53.866315+0000","last_became_active":"2024-03-04T16:20:07.484919+0000","last_became_peered":"2024-03-04T16:20:07.484919+0000","last_unstale":"2024-03-04T16:23:53.866315+0000","last_undegraded":"2024-03-04T16:23:53.866315+0000","last_fullsized":"2024-03-04T16:23:53.866315+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:22:08.721229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:08.721229+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048817+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"453'10022","reported_seq":10136,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:52.814311+0000","last_change":"2024-03-04T16:21:25.449756+0000","last_active":"2024-03-04T16:23:52.814311+0000","last_peered":"2024-03-04T16:23:52.814311+0000","last_clean":"2024-03-04T16:23:52.814311+0000","last_became_active":"2024-03-04T16:21:25.449469+0000","last_became_peered":"2024-03-04T16:21:25.449469+0000","last_unstale":"2024-03-04T16:23:52.814311+0000","last_undegraded":"2024-03-04T16:23:52.814311+0000","last_fullsized":"2024-03-04T16:23:52.814311+0000","mapping_epoch":401,"log_start":"439'5300","ondisk_log_start":"439'5300","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.284795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231997440,"num_objects":3540,"num_object_clones":0,"num_object_copies":10620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10022,"num_write_kb":433984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":808,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.870049+0000","last_change":"2024-03-04T16:23:04.878400+0000","last_active":"2024-03-04T16:23:53.870049+0000","last_peered":"2024-03-04T16:23:53.870049+0000","last_clean":"2024-03-04T16:23:53.870049+0000","last_became_active":"2024-03-04T16:21:45.424326+0000","last_became_peered":"2024-03-04T16:21:45.424326+0000","last_unstale":"2024-03-04T16:23:53.870049+0000","last_undegraded":"2024-03-04T16:23:53.870049+0000","last_fullsized":"2024-03-04T16:23:53.870049+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.878278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.878278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:30.211352+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":309,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.837604+0000","last_change":"2024-03-04T16:22:42.336778+0000","last_active":"2024-03-04T16:23:50.837604+0000","last_peered":"2024-03-04T16:23:50.837604+0000","last_clean":"2024-03-04T16:23:50.837604+0000","last_became_active":"2024-03-04T16:21:25.438797+0000","last_became_peered":"2024-03-04T16:21:25.438797+0000","last_unstale":"2024-03-04T16:23:50.837604+0000","last_undegraded":"2024-03-04T16:23:50.837604+0000","last_fullsized":"2024-03-04T16:23:50.837604+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:42.336656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.336656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:02.043125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"441'42","reported_seq":961,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.866184+0000","last_change":"2024-03-04T16:23:07.898360+0000","last_active":"2024-03-04T16:23:53.866184+0000","last_peered":"2024-03-04T16:23:53.866184+0000","last_clean":"2024-03-04T16:23:53.866184+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:23:53.866184+0000","last_undegraded":"2024-03-04T16:23:53.866184+0000","last_fullsized":"2024-03-04T16:23:53.866184+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-04T16:23:07.898070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.898070+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:32.268466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":199,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.867390+0000","last_change":"2024-03-04T16:22:43.866317+0000","last_active":"2024-03-04T16:23:53.867390+0000","last_peered":"2024-03-04T16:23:53.867390+0000","last_clean":"2024-03-04T16:23:53.867390+0000","last_became_active":"2024-03-04T16:20:07.487202+0000","last_became_peered":"2024-03-04T16:20:07.487202+0000","last_unstale":"2024-03-04T16:23:53.867390+0000","last_undegraded":"2024-03-04T16:23:53.867390+0000","last_fullsized":"2024-03-04T16:23:53.867390+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.866200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.866200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:54.048873+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"454'9851","reported_seq":10027,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:54.414901+0000","last_change":"2024-03-04T16:22:42.067136+0000","last_active":"2024-03-04T16:23:54.414901+0000","last_peered":"2024-03-04T16:23:54.414901+0000","last_clean":"2024-03-04T16:23:54.414901+0000","last_became_active":"2024-03-04T16:21:25.450165+0000","last_became_peered":"2024-03-04T16:21:25.450165+0000","last_unstale":"2024-03-04T16:23:54.414901+0000","last_undegraded":"2024-03-04T16:23:54.414901+0000","last_fullsized":"2024-03-04T16:23:54.414901+0000","mapping_epoch":401,"log_start":"441'5600","ondisk_log_start":"441'5600","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5014","last_scrub_stamp":"2024-03-04T16:22:42.066893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.066893+0000","objects_scrubbed":4968,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.270207+0000","scrub_duration":2144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222887936,"num_objects":3401,"num_object_clones":0,"num_object_copies":10203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9851,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":845,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.834884+0000","last_change":"2024-03-04T16:22:46.243647+0000","last_active":"2024-03-04T16:23:50.834884+0000","last_peered":"2024-03-04T16:23:50.834884+0000","last_clean":"2024-03-04T16:23:50.834884+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:23:50.834884+0000","last_undegraded":"2024-03-04T16:23:50.834884+0000","last_fullsized":"2024-03-04T16:23:50.834884+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.243413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.243413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:51.507147+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"366'9","reported_seq":271,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.835475+0000","last_change":"2024-03-04T16:23:02.057648+0000","last_active":"2024-03-04T16:23:50.835475+0000","last_peered":"2024-03-04T16:23:50.835475+0000","last_clean":"2024-03-04T16:23:50.835475+0000","last_became_active":"2024-03-04T16:20:07.484334+0000","last_became_peered":"2024-03-04T16:20:07.484334+0000","last_unstale":"2024-03-04T16:23:50.835475+0000","last_undegraded":"2024-03-04T16:23:50.835475+0000","last_fullsized":"2024-03-04T16:23:50.835475+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:23:02.057434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.057434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.009367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":848,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.866103+0000","last_change":"2024-03-04T16:22:48.920668+0000","last_active":"2024-03-04T16:23:53.866103+0000","last_peered":"2024-03-04T16:23:53.866103+0000","last_clean":"2024-03-04T16:23:53.866103+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:23:53.866103+0000","last_undegraded":"2024-03-04T16:23:53.866103+0000","last_fullsized":"2024-03-04T16:23:53.866103+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.920550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.920550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:58.681425+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'46","reported_seq":942,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.881837+0000","last_change":"2024-03-04T16:23:00.358198+0000","last_active":"2024-03-04T16:23:53.881837+0000","last_peered":"2024-03-04T16:23:53.881837+0000","last_clean":"2024-03-04T16:23:53.881837+0000","last_became_active":"2024-03-04T16:20:07.507258+0000","last_became_peered":"2024-03-04T16:20:07.507258+0000","last_unstale":"2024-03-04T16:23:53.881837+0000","last_undegraded":"2024-03-04T16:23:53.881837+0000","last_fullsized":"2024-03-04T16:23:53.881837+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:23:00.358056+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.358056+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.797848+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"455'10274","reported_seq":12206,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:57.626467+0000","last_change":"2024-03-04T16:21:56.410088+0000","last_active":"2024-03-04T16:23:57.626467+0000","last_peered":"2024-03-04T16:23:57.626467+0000","last_clean":"2024-03-04T16:23:57.626467+0000","last_became_active":"2024-03-04T16:21:56.407370+0000","last_became_peered":"2024-03-04T16:21:56.407370+0000","last_unstale":"2024-03-04T16:23:57.626467+0000","last_undegraded":"2024-03-04T16:23:57.626467+0000","last_fullsized":"2024-03-04T16:23:57.626467+0000","mapping_epoch":418,"log_start":"439'5400","ondisk_log_start":"439'5400","created":401,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213778432,"num_objects":3262,"num_object_clones":0,"num_object_copies":9786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10274,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1068,"num_bytes_recovered":69992448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":115,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.477104+0000","last_change":"2024-03-04T16:23:00.740759+0000","last_active":"2024-03-04T16:23:56.477104+0000","last_peered":"2024-03-04T16:23:56.477104+0000","last_clean":"2024-03-04T16:23:56.477104+0000","last_became_active":"2024-03-04T16:21:45.420505+0000","last_became_peered":"2024-03-04T16:21:45.420505+0000","last_unstale":"2024-03-04T16:23:56.477104+0000","last_undegraded":"2024-03-04T16:23:56.477104+0000","last_fullsized":"2024-03-04T16:23:56.477104+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:23:00.740631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.740631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.137776+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":869,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.836050+0000","last_change":"2024-03-04T16:23:12.207020+0000","last_active":"2024-03-04T16:23:50.836050+0000","last_peered":"2024-03-04T16:23:50.836050+0000","last_clean":"2024-03-04T16:23:50.836050+0000","last_became_active":"2024-03-04T16:21:45.425391+0000","last_became_peered":"2024-03-04T16:21:45.425391+0000","last_unstale":"2024-03-04T16:23:50.836050+0000","last_undegraded":"2024-03-04T16:23:50.836050+0000","last_fullsized":"2024-03-04T16:23:50.836050+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.206943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.206943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:38.403715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":806,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.852443+0000","last_change":"2024-03-04T16:22:06.694118+0000","last_active":"2024-03-04T16:23:50.852443+0000","last_peered":"2024-03-04T16:23:50.852443+0000","last_clean":"2024-03-04T16:23:50.852443+0000","last_became_active":"2024-03-04T16:20:07.471591+0000","last_became_peered":"2024-03-04T16:20:07.471591+0000","last_unstale":"2024-03-04T16:23:50.852443+0000","last_undegraded":"2024-03-04T16:23:50.852443+0000","last_fullsized":"2024-03-04T16:23:50.852443+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.693989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.693989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057437+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"441'32","reported_seq":135,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.475856+0000","last_change":"2024-03-04T16:23:05.801666+0000","last_active":"2024-03-04T16:23:56.475856+0000","last_peered":"2024-03-04T16:23:56.475856+0000","last_clean":"2024-03-04T16:23:56.475856+0000","last_became_active":"2024-03-04T16:21:45.446219+0000","last_became_peered":"2024-03-04T16:21:45.446219+0000","last_unstale":"2024-03-04T16:23:56.475856+0000","last_undegraded":"2024-03-04T16:23:56.475856+0000","last_fullsized":"2024-03-04T16:23:56.475856+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:23:05.801531+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:23:05.801531+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:07.742131+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"453'10045","reported_seq":10233,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.085281+0000","last_change":"2024-03-04T16:22:58.382258+0000","last_active":"2024-03-04T16:23:53.085281+0000","last_peered":"2024-03-04T16:23:53.085281+0000","last_clean":"2024-03-04T16:23:53.085281+0000","last_became_active":"2024-03-04T16:21:25.445028+0000","last_became_peered":"2024-03-04T16:21:25.445028+0000","last_unstale":"2024-03-04T16:23:53.085281+0000","last_undegraded":"2024-03-04T16:23:53.085281+0000","last_fullsized":"2024-03-04T16:23:53.085281+0000","mapping_epoch":401,"log_start":"435'4400","ondisk_log_start":"435'4400","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6281","last_scrub_stamp":"2024-03-04T16:22:58.381858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.381858+0000","objects_scrubbed":6184,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.638463+0000","scrub_duration":2894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233897984,"num_objects":3569,"num_object_clones":0,"num_object_copies":10707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10045,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":271,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.849518+0000","last_change":"2024-03-04T16:22:10.868223+0000","last_active":"2024-03-04T16:23:50.849518+0000","last_peered":"2024-03-04T16:23:50.849518+0000","last_clean":"2024-03-04T16:23:50.849518+0000","last_became_active":"2024-03-04T16:21:45.423031+0000","last_became_peered":"2024-03-04T16:21:45.423031+0000","last_unstale":"2024-03-04T16:23:50.849518+0000","last_undegraded":"2024-03-04T16:23:50.849518+0000","last_fullsized":"2024-03-04T16:23:50.849518+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:10.867960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:10.867960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:53.057467+0000","scrub_duration":246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":860,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876393+0000","last_change":"2024-03-04T16:21:46.008458+0000","last_active":"2024-03-04T16:23:53.876393+0000","last_peered":"2024-03-04T16:23:53.876393+0000","last_clean":"2024-03-04T16:23:53.876393+0000","last_became_active":"2024-03-04T16:21:45.434815+0000","last_became_peered":"2024-03-04T16:21:45.434815+0000","last_unstale":"2024-03-04T16:23:53.876393+0000","last_undegraded":"2024-03-04T16:23:53.876393+0000","last_fullsized":"2024-03-04T16:23:53.876393+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:46.008376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:21:46.008376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":112,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:56.476614+0000","last_change":"2024-03-04T16:23:01.769342+0000","last_active":"2024-03-04T16:23:56.476614+0000","last_peered":"2024-03-04T16:23:56.476614+0000","last_clean":"2024-03-04T16:23:56.476614+0000","last_became_active":"2024-03-04T16:21:45.423438+0000","last_became_peered":"2024-03-04T16:21:45.423438+0000","last_unstale":"2024-03-04T16:23:56.476614+0000","last_undegraded":"2024-03-04T16:23:56.476614+0000","last_fullsized":"2024-03-04T16:23:56.476614+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:01.769222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:23:01.769222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:05.705541+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":964,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.851091+0000","last_change":"2024-03-04T16:23:10.470966+0000","last_active":"2024-03-04T16:23:50.851091+0000","last_peered":"2024-03-04T16:23:50.851091+0000","last_clean":"2024-03-04T16:23:50.851091+0000","last_became_active":"2024-03-04T16:21:45.439301+0000","last_became_peered":"2024-03-04T16:21:45.439301+0000","last_unstale":"2024-03-04T16:23:50.851091+0000","last_undegraded":"2024-03-04T16:23:50.851091+0000","last_fullsized":"2024-03-04T16:23:50.851091+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-04T16:23:10.470699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.470699+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.738180+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"455'10292","reported_seq":12036,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-04T16:23:57.589047+0000","last_change":"2024-03-04T16:21:59.474717+0000","last_active":"2024-03-04T16:23:57.589047+0000","last_peered":"2024-03-04T16:23:57.589047+0000","last_clean":"2024-03-04T16:23:57.589047+0000","last_became_active":"2024-03-04T16:21:59.471792+0000","last_became_peered":"2024-03-04T16:21:59.471792+0000","last_unstale":"2024-03-04T16:23:57.589047+0000","last_undegraded":"2024-03-04T16:23:57.589047+0000","last_fullsized":"2024-03-04T16:23:57.589047+0000","mapping_epoch":421,"log_start":"439'5400","ondisk_log_start":"439'5400","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211812352,"num_objects":3232,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10292,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":70123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":796,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.876015+0000","last_change":"2024-03-04T16:22:44.238092+0000","last_active":"2024-03-04T16:23:53.876015+0000","last_peered":"2024-03-04T16:23:53.876015+0000","last_clean":"2024-03-04T16:23:53.876015+0000","last_became_active":"2024-03-04T16:20:07.480213+0000","last_became_peered":"2024-03-04T16:20:07.480213+0000","last_unstale":"2024-03-04T16:23:53.876015+0000","last_undegraded":"2024-03-04T16:23:53.876015+0000","last_fullsized":"2024-03-04T16:23:53.876015+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.237641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.237641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:23:57.682755+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":321,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.851186+0000","last_change":"2024-03-04T16:22:58.485899+0000","last_active":"2024-03-04T16:23:50.851186+0000","last_peered":"2024-03-04T16:23:50.851186+0000","last_clean":"2024-03-04T16:23:50.851186+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:23:50.851186+0000","last_undegraded":"2024-03-04T16:23:50.851186+0000","last_fullsized":"2024-03-04T16:23:50.851186+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:58.485608+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.485608+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:08.537438+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":132,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-04T16:23:50.847959+0000","last_change":"2024-03-04T16:23:12.437298+0000","last_active":"2024-03-04T16:23:50.847959+0000","last_peered":"2024-03-04T16:23:50.847959+0000","last_clean":"2024-03-04T16:23:50.847959+0000","last_became_active":"2024-03-04T16:21:24.444615+0000","last_became_peered":"2024-03-04T16:21:24.444615+0000","last_unstale":"2024-03-04T16:23:50.847959+0000","last_undegraded":"2024-03-04T16:23:50.847959+0000","last_fullsized":"2024-03-04T16:23:50.847959+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.437183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.437183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.188381+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":848,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-04T16:23:53.864474+0000","last_change":"2024-03-04T16:22:56.926525+0000","last_active":"2024-03-04T16:23:53.864474+0000","last_peered":"2024-03-04T16:23:53.864474+0000","last_clean":"2024-03-04T16:23:53.864474+0000","last_became_active":"2024-03-04T16:21:45.425802+0000","last_became_peered":"2024-03-04T16:21:45.425802+0000","last_unstale":"2024-03-04T16:23:53.864474+0000","last_undegraded":"2024-03-04T16:23:53.864474+0000","last_fullsized":"2024-03-04T16:23:53.864474+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:22:56.926373+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.926373+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:03.476141+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3576889368,"num_objects":54580,"num_object_clones":0,"num_object_copies":163740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54580,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159694,"num_write_kb":6856705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8309,"num_bytes_recovered":544538624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10767577088,"data_stored":10767564872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76394,"ondisk_log_size":76394,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1261,"num_read_kb":1256,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":8906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070722,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2521488,"kb_used_data":1601796,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":91223664,"statfs":{"total":95995035648,"available":93413031936,"internally_reserved":0,"allocated":1640239104,"data_stored":1639820454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":941750730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365063,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2083872,"kb_used_data":1677848,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91661280,"statfs":{"total":95995035648,"available":93861150720,"internally_reserved":0,"allocated":1718116352,"data_stored":1717745250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575136,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3109932,"kb_used_data":1991712,"kb_used_omap":1,"kb_used_meta":1118206,"kb_avail":90635220,"statfs":{"total":95995035648,"available":92810465280,"internally_reserved":0,"allocated":2039513088,"data_stored":2039064422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1145043397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738660,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451784,"kb_used_data":1554304,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91293368,"statfs":{"total":95995035648,"available":93484408832,"internally_reserved":0,"allocated":1591607296,"data_stored":1591192084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411610,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1937396,"kb_used_data":1350892,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91807756,"statfs":{"total":95995035648,"available":94011142144,"internally_reserved":0,"allocated":1383313408,"data_stored":1382917176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149973,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2724328,"kb_used_data":1544540,"kb_used_omap":1,"kb_used_meta":1179646,"kb_avail":91020824,"statfs":{"total":95995035648,"available":93205323776,"internally_reserved":0,"allocated":1581608960,"data_stored":1581171800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1207957960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626700,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1926200,"kb_used_data":906480,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":91818952,"statfs":{"total":95995035648,"available":94022606848,"internally_reserved":0,"allocated":928235520,"data_stored":927834994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1044117958},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1568735232,"data_stored":1568735232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2006257664,"data_stored":2006253592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1380519936,"data_stored":1380515864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1569718272,"data_stored":1569718272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":924647424,"data_stored":924647424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1694437376,"data_stored":1694433304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1623261184,"data_stored":1623261184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:24:00.169 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-04T16:24:00.170 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:24:00.170 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-04T16:24:00.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-04T16:24:00.198 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:00.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:00.329 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:00.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: from='client.35990 v1:172.21.15.38:0/27119184' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:24:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: osdmap e456: 8 total, 8 up, 7 in 2024-03-04T16:24:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:24:00.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:23:59 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:24:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.359 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:00.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.488 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:00.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:00.636 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:00.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:00.738 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:00.768 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:00.870 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:00.901 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:00.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:01.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:01.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.175 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:01.242 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:00 smithi038 ceph-mon[38113]: pgmap v998: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:01.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:00 smithi038 ceph-mon[29779]: pgmap v998: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:01.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:01.309 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:01.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:00 smithi176 ceph-mon[33619]: pgmap v998: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:01.411 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.436 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:01.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:01.574 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:01.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:01.717 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:01.819 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.848 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:01.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:01.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:01.989 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:01.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:02.090 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:02.117 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:02.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:02.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:01 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:24:02.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:01 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:24:02.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:01 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:24:02.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:01 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:24:02.260 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:02.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:01 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:24:02.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:01 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:24:02.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:02.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:02.494 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:02.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:02.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:02.688 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:02.791 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:02.821 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:02.923 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:02.972 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:02.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:03.074 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:03.087 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: pgmap v999: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[38113]: 11.1 scrub starts 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:24:03.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:24:03.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: pgmap v999: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:24:03.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:02 smithi038 ceph-mon[29779]: 11.1 scrub starts 2024-03-04T16:24:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.102 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:03.203 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.235 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:03.258 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:24:03.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:24:03.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:24:03.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:24:03.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: pgmap v999: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 648 B/s rd, 0 B/s wr, 925 op/s 2024-03-04T16:24:03.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:02 smithi176 ceph-mon[33619]: 11.1 scrub starts 2024-03-04T16:24:03.337 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:03.365 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:03.452 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:03.467 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:03.494 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:03.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:03.631 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:03.733 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:03.760 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:03.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:03.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:03.862 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:03.879 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:24:03.879 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 20, 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 600, 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "ops": [ 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:223776 10.f 10:f9b98c7b:::benchmark_data_smithi038_155391_object87819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:06.468793+0000", 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 236.97469153599999, 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.552882903, 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 223776 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468793+0000", 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.881 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468793+0000", 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468795+0000", 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4440000000000001e-06 2024-03-04T16:24:03.882 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468800+0000", 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9069999999999998e-06 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468808+0000", 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4349999999999997e-06 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.883 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.468815+0000", 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4130000000000003e-06 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.470209+0000", 2024-03-04T16:24:03.884 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001394492 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.470363+0000", 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015366300000000001 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2,4", 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.470519+0000", 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000155814 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.885 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.513472+0000", 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.042953571000000003 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:06.513482+0000", 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0176e-05 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.498781+0000", 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.98529880700000005 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.886 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:07.498792+0000", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1286e-05 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.719836+0000", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2210432039999999 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:08.720082+0000", 2024-03-04T16:24:03.887 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00024683700000000002 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:09.021449+0000", 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.30136672599999997 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:09.021676+0000", 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000227128 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:227938 10.f 10:ff830b29:::benchmark_data_smithi038_155391_object91981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.364299+0000", 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 228.07918563300001, 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.116739734, 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227938 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.889 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364299+0000", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364299+0000", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364301+0000", 2024-03-04T16:24:03.890 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.345e-06 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364305+0000", 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.0350000000000003e-06 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364312+0000", 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2189999999999998e-06 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.891 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364319+0000", 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6599999999999998e-06 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364493+0000", 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000174816 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364607+0000", 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114076 2024-03-04T16:24:03.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2,4", 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.364724+0000", 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011691 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.370198+0000", 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005473838 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.893 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.380803+0000", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010604792 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392116+0000", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011313222 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392131+0000", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4707e-05 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.894 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.480964+0000", 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0888330589999999 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.481039+0000", 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.5055000000000006e-05 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.895 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:227947 10.f 10:f4e10788:::benchmark_data_smithi038_155391_object91990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.376890+0000", 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 228.06659448600001, 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1289437, 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "reached pg", 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227947 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.896 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376890+0000", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376890+0000", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376892+0000", 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4219999999999999e-06 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.897 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376897+0000", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3079999999999998e-06 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376905+0000", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.191e-06 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.376913+0000", 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3309999999999998e-06 2024-03-04T16:24:03.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.381303+0000", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0043903040000000003 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.381426+0000", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012268500000000001 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2,4", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.381557+0000", 2024-03-04T16:24:03.899 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130944 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392165+0000", 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010608618 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392176+0000", 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0380000000000001e-05 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.900 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.490996+0000", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.098820763 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for missing object", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.491110+0000", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113727 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505720+0000", 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014609903 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.901 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505834+0000", 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011412399999999999 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:227949 10.f 10:f4688b8f:::benchmark_data_smithi038_155391_object91992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.379683+0000", 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 228.06380178399999, 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.131532314, 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227949 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379683+0000", 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.903 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379683+0000", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379685+0000", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.652e-06 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379690+0000", 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0860000000000001e-06 2024-03-04T16:24:03.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379700+0000", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0032e-05 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.379708+0000", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8900000000000007e-06 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392206+0000", 2024-03-04T16:24:03.905 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012498514 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.392215+0000", 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8580000000000006e-06 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.491136+0000", 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0989203599999999 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.906 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.491237+0000", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010186799999999999 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.491360+0000", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012214300000000001 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505293+0000", 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.013933399000000001 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.907 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505308+0000", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4688000000000001e-05 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511050+0000", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005742278 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511102+0000", 2024-03-04T16:24:03.908 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.223e-05 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511215+0000", 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113316 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:227954 10.f 10:f773d4b6:::benchmark_data_smithi038_155391_object91997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.397473+0000", 2024-03-04T16:24:03.909 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 228.04601211100001, 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1139371810000001, 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227954 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397473+0000", 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397473+0000", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397473+0000", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2799999999999996e-07 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397476+0000", 2024-03-04T16:24:03.911 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9660000000000001e-06 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397482+0000", 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6180000000000003e-06 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.397487+0000", 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2490000000000001e-06 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.912 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.407673+0000", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010185654000000001 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.407685+0000", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1785e-05 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.493594+0000", 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0859089980000001 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.913 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.493722+0000", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012830799999999999 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.493856+0000", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013423800000000001 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505314+0000", 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.011457288 2024-03-04T16:24:03.914 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505373+0000", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.9614000000000002e-05 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511261+0000", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005887672 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511312+0000", 2024-03-04T16:24:03.915 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0693999999999999e-05 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511410+0000", 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.8468999999999995e-05 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:227957 10.f 10:f8ef3de5:::benchmark_data_smithi038_155391_object92000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.400013+0000", 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 228.04347237900001, 2024-03-04T16:24:03.916 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.111600752, 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 227957 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400013+0000", 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.917 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400013+0000", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400014+0000", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6199999999999999e-06 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400019+0000", 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.7559999999999999e-06 2024-03-04T16:24:03.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400030+0000", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0883000000000001e-05 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.400037+0000", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.99e-06 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.407709+0000", 2024-03-04T16:24:03.919 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076723099999999999 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.407717+0000", 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.7880000000000007e-06 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.496078+0000", 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.088361168 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.920 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.496188+0000", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000110352 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.496309+0000", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000120803 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505319+0000", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0090100600000000003 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.921 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505884+0000", 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00056432499999999996 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511458+0000", 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0055746600000000004 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511514+0000", 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5572000000000003e-05 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.922 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511613+0000", 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9464999999999997e-05 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228001 10.f 10:f27e0cab:::benchmark_data_smithi038_155391_object92044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.451094+0000", 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.99239055699999, 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.060729466, 2024-03-04T16:24:03.923 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228001 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451094+0000", 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.924 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451094+0000", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451096+0000", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.9139999999999998e-06 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451102+0000", 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5210000000000002e-06 2024-03-04T16:24:03.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451115+0000", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3217999999999999e-05 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.451123+0000", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.9209999999999992e-06 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.452560+0000", 2024-03-04T16:24:03.926 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0014374419999999999 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.452571+0000", 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0594000000000001e-05 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.498296+0000", 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0457245420000001 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.927 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.498409+0000", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000113198 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.498536+0000", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012745200000000001 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505325+0000", 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0067890210000000001 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.928 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505930+0000", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00060520599999999995 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511669+0000", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.005738099 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511724+0000", 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5903000000000003e-05 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.929 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511824+0000", 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9434999999999999e-05 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228004 10.f 10:fc18ec07:::benchmark_data_smithi038_155391_object92047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.455785+0000", 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.98770034099999, 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.06086671, 2024-03-04T16:24:03.930 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228004 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455785+0000", 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.931 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455785+0000", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455786+0000", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.6750000000000001e-06 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455792+0000", 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7760000000000003e-06 2024-03-04T16:24:03.932 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455801+0000", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4450000000000002e-06 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455838+0000", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6211999999999998e-05 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455944+0000", 2024-03-04T16:24:03.933 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000106458 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.455962+0000", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7717e-05 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.500539+0000", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.044577485 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.500662+0000", 2024-03-04T16:24:03.934 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012245600000000001 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.500789+0000", 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012745899999999999 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505331+0000", 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045412380000000004 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.935 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.506194+0000", 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00086349300000000001 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516493+0000", 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.010298705999999999 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516543+0000", 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0413e-05 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.936 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516651+0000", 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000108177 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228011 10.f 10:ff46dc25:::benchmark_data_smithi038_155391_object92054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.457546+0000", 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.98593924900001, 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.071563123, 2024-03-04T16:24:03.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228011 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457546+0000", 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457546+0000", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457547+0000", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1289999999999999e-06 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457555+0000", 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0150000000000003e-06 2024-03-04T16:24:03.939 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457562+0000", 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1250000000000004e-06 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457567+0000", 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.5149999999999997e-06 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.940 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457625+0000", 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7428e-05 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.457635+0000", 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.9839999999999996e-06 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.502911+0000", 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.045276001 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.941 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.503140+0000", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00022879500000000001 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.503275+0000", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013562900000000001 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.505833+0000", 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0025577220000000001 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.942 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511000+0000", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0051665590000000003 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.528946+0000", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0179461 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529008+0000", 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.2161000000000003e-05 2024-03-04T16:24:03.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529109+0000", 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010096 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228035 10.f 10:f14eca3d:::benchmark_data_smithi038_155391_object92078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.492869+0000", 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.95061587800001, 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0364265589999999, 2024-03-04T16:24:03.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228035 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492869+0000", 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.945 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492869+0000", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492871+0000", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.8589999999999999e-06 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492877+0000", 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.6579999999999999e-06 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.946 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492888+0000", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1873e-05 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.492896+0000", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.0260000000000007e-06 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.493291+0000", 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00039474399999999999 2024-03-04T16:24:03.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.493306+0000", 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4552999999999999e-05 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.506239+0000", 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0129328150000001 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.948 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.506364+0000", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012532400000000001 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.506508+0000", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014365600000000001 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516454+0000", 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0099461299999999992 2024-03-04T16:24:03.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516661+0000", 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020769300000000001 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529154+0000", 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012492826 2024-03-04T16:24:03.950 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529207+0000", 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.2605000000000002e-05 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529296+0000", 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8796999999999997e-05 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.951 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228039 10.f 10:f67daf6e:::benchmark_data_smithi038_155391_object92082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.496938+0000", 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.946546891, 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0325339090000001, 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.952 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228039 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496938+0000", 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496938+0000", 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496940+0000", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.793e-06 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496945+0000", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3480000000000003e-06 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496960+0000", 2024-03-04T16:24:03.954 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4898e-05 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.496968+0000", 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4930000000000001e-06 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.497138+0000", 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000170695 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.955 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.497149+0000", 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0742e-05 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.508711+0000", 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0115623949999999 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.956 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.508835+0000", 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000123614 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.508981+0000", 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014604099999999999 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.957 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516667+0000", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0076858009999999999 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516745+0000", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.8039000000000003e-05 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529332+0000", 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.012587429000000001 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.958 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529383+0000", 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0348e-05 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.529472+0000", 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9272999999999997e-05 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228052 10.f 10:fb0967d8:::benchmark_data_smithi038_155391_object92095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.959 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.513187+0000", 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.93029753900001, 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0249662369999999, 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228052 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.960 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513187+0000", 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513187+0000", 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513189+0000", 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.7880000000000001e-06 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513194+0000", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0209999999999999e-06 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513203+0000", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9989999999999997e-06 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513211+0000", 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3220000000000004e-06 2024-03-04T16:24:03.962 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513368+0000", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00015782200000000001 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.513381+0000", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2265e-05 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511861+0000", 2024-03-04T16:24:03.963 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.99848045900000004 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.511986+0000", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012483199999999999 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.512117+0000", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013093600000000001 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516672+0000", 2024-03-04T16:24:03.964 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045555910000000003 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516779+0000", 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00010705 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538058+0000", 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.021278845000000001 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.965 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538110+0000", 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.1183000000000001e-05 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538154+0000", 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4124e-05 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228088 10.f 10:f3d36a32:::benchmark_data_smithi038_155391_object92131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.562266+0000", 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.881219096, 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97601007500000003, 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.967 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228088 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562266+0000", 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562266+0000", 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.968 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562268+0000", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.075e-06 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562274+0000", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.105e-06 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562288+0000", 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.4107000000000001e-05 2024-03-04T16:24:03.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562296+0000", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.4800000000000004e-06 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562361+0000", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5198999999999997e-05 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.562371+0000", 2024-03-04T16:24:03.970 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0369e-05 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.514181+0000", 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95181025399999997 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.514305+0000", 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000123779 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.971 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.514443+0000", 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00013765500000000001 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516713+0000", 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022705830000000001 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.519641+0000", 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0029275310000000001 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.972 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538192+0000", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.018551020000000001 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538238+0000", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5630000000000002e-05 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538276+0000", 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8287999999999997e-05 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.973 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:228094 10.f 10:fb3fd1e5:::benchmark_data_smithi038_155391_object92137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:15.573276+0000", 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 227.87020898099999, 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96528712400000005, 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 228094 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.974 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573276+0000", 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573276+0000", 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.975 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573278+0000", 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.86e-06 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573284+0000", 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0549999999999996e-06 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573295+0000", 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1214e-05 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.976 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573302+0000", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.2919999999999998e-06 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573504+0000", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00020144499999999999 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:15.573515+0000", 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1438e-05 2024-03-04T16:24:03.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516806+0000", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94329058300000002 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.516913+0000", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000107421 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.517040+0000", 2024-03-04T16:24:03.978 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012652400000000001 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.524174+0000", 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0071337520000000001 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.524186+0000", 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2897999999999999e-05 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.979 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538319+0000", 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.014132135000000001 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538509+0000", 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00019043200000000001 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:16.538563+0000", 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4075e-05 2024-03-04T16:24:03.980 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253594 10.17 10:ef25dcca:::benchmark_data_smithi038_155391_object117637:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.484989+0000", 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.95849594399999, 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.97265980799999996, 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.981 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253594 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.484989+0000", 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.484989+0000", 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.484991+0000", 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.2850000000000001e-06 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.484998+0000", 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.6100000000000002e-06 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.983 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.485011+0000", 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.2889e-05 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.485020+0000", 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.9409999999999996e-06 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.485084+0000", 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4046000000000003e-05 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.984 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.485093+0000", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.6549999999999999e-06 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.453288+0000", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96819489999999997 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.453386+0000", 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.7532000000000001e-05 2024-03-04T16:24:03.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.453510+0000", 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012371100000000001 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457518+0000", 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0040084439999999999 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.986 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457544+0000", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.6268999999999999e-05 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457574+0000", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 2.9547e-05 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457611+0000", 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7441999999999998e-05 2024-03-04T16:24:03.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457649+0000", 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.7537000000000003e-05 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253600 10.17 10:e8909601:::benchmark_data_smithi038_155391_object117643:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.493182+0000", 2024-03-04T16:24:03.988 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.950302792, 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96852095199999999, 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253600 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493182+0000", 2024-03-04T16:24:03.989 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493182+0000", 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493184+0000", 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.601e-06 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.990 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493189+0000", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.9980000000000003e-06 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493198+0000", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.3749999999999992e-06 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493206+0000", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.486e-06 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.991 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493266+0000", 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0615999999999997e-05 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.493276+0000", 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.4509999999999998e-06 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.455732+0000", 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.96245610599999998 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.992 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.455862+0000", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000130016 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.456010+0000", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014794499999999999 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457757+0000", 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0017467909999999999 2024-03-04T16:24:03.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461561+0000", 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0038046899999999999 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461603+0000", 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.1801000000000002e-05 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:03.994 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461657+0000", 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.4203999999999998e-05 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461703+0000", 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.5871999999999997e-05 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.995 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253618 10.17 10:e8a788f7:::benchmark_data_smithi038_155391_object117661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.512320+0000", 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.931164792, 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.95192400499999996, 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253618 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.996 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512320+0000", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512320+0000", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512322+0000", 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.519e-06 2024-03-04T16:24:03.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512327+0000", 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.728e-06 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512338+0000", 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1022999999999999e-05 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:03.998 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512347+0000", 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.4020000000000002e-06 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512407+0000", 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0319000000000003e-05 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.512416+0000", 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 9.0790000000000007e-06 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:03.999 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457788+0000", 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94537199299999997 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.457899+0000", 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00011099000000000001 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:04.000 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.458012+0000", 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000112316 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461776+0000", 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0037641620000000002 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464052+0000", 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0022763309999999999 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.001 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464140+0000", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8199e-05 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464194+0000", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.3538000000000001e-05 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464244+0000", 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.0405999999999998e-05 2024-03-04T16:24:04.002 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253626 10.17 10:ef6783bc:::benchmark_data_smithi038_155391_object117669:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.521910+0000", 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.92157495399999, 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.94245566000000003, 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253626 2024-03-04T16:24:04.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521910+0000", 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:04.004 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521910+0000", 2024-03-04T16:24:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:04.007 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521911+0000", 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.1800000000000005e-07 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521914+0000", 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.18e-06 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521920+0000", 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.5010000000000001e-06 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.008 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.521926+0000", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.0900000000000001e-06 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.528596+0000", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.006669917 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.528605+0000", 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.1859999999999992e-06 2024-03-04T16:24:04.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.459675+0000", 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93107081800000002 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.459790+0000", 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114425 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:04.010 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.459919+0000", 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012967999999999999 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464096+0000", 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0041765259999999998 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464108+0000", 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1515000000000001e-05 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.011 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464285+0000", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000177016 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464330+0000", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.4953999999999997e-05 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464366+0000", 2024-03-04T16:24:04.012 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.6134000000000003e-05 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253636 10.17 10:ece50722:::benchmark_data_smithi038_155391_object117679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.531989+0000", 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.91149611500001, 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.93838716899999997, 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:04.013 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253636 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.531989+0000", 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.531989+0000", 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.014 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.531990+0000", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.641e-06 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.531996+0000", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 5.7910000000000001e-06 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.532008+0000", 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.1651000000000001e-05 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.015 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.532016+0000", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.3699999999999995e-06 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.533017+0000", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.001000304 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.533024+0000", 2024-03-04T16:24:04.016 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.3819999999999998e-06 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461838+0000", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.92881414200000001 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.461986+0000", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00014761700000000001 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.462112+0000", 2024-03-04T16:24:04.017 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00012651900000000001 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.467018+0000", 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0049054110000000001 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.467060+0000", 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.2114999999999999e-05 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:04.018 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470265+0000", 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0032050360000000001 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470329+0000", 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 6.4220999999999994e-05 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470376+0000", 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.6968999999999997e-05 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:04.019 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "description": "osd_op(client.35585.0:253698 10.17 10:e9f26597:::benchmark_data_smithi038_155391_object117741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e389)", 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "initiated_at": "2024-03-04T16:20:47.584182+0000", 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "age": 195.859302464, 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.88627978799999996, 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "type_data": { 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_info": { 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client": "client.35585", 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tid": 253698 2024-03-04T16:24:04.020 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "events": [ 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "initiated", 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584182+0000", 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "header_read", 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584182+0000", 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "throttled", 2024-03-04T16:24:04.021 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584184+0000", 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.3629999999999999e-06 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "all_read", 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584189+0000", 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 4.8779999999999997e-06 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "dispatched", 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584199+0000", 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 1.0261000000000001e-05 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.022 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "queued_for_pg", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.584207+0000", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 7.6869999999999994e-06 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.588723+0000", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0045161389999999997 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for peered", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:47.588732+0000", 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 8.8789999999999992e-06 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.023 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "reached_pg", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464395+0000", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.87566306699999996 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "started", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464547+0000", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000152268 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.464661+0000", 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.000114002 2024-03-04T16:24:04.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.467094+0000", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0024326069999999998 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470132+0000", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.0030382069999999998 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "op_commit", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470388+0000", 2024-03-04T16:24:04.025 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 0.00025657700000000002 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "commit_sent", 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470427+0000", 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.8313999999999999e-05 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "event": "done", 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "time": "2024-03-04T16:20:48.470462+0000", 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: "duration": 3.5539e-05 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.026 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:04.027 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:04.027 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:24:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.028 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:04.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:04.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:04.156 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:04.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[29779]: 11.1 scrub ok 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[38113]: 11.1 scrub ok 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:24:04.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:03 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:24:04.258 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:04.288 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:24:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:24:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-mon[33619]: 11.1 scrub ok 2024-03-04T16:24:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:24:04.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:03 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:24:04.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.424 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:04.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:04.525 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:04.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:04.656 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:04.694 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:04.796 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:04.833 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:04.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:04.963 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:04.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:05.065 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:05.102 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:05.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:24:05.204 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:05.239 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:05.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:05.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:04 smithi038 ceph-mon[29779]: pgmap v1000: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-04T16:24:05.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:05.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:04 smithi038 ceph-mon[38113]: pgmap v1000: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-04T16:24:05.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:05.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:05.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:04 smithi176 ceph-mon[33619]: pgmap v1000: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-04T16:24:05.384 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:05.428 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:05.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:05.519 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:05.522 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:05.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:05.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.697 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:05.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.849 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:05.950 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:05.995 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:05.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:06.097 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:06.135 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:06.227 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:24:06.227 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:24:06.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:24:06.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:24:06.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:24:06.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2311402469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:05 smithi176 ceph-mon[33619]: from='client.25921 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.237 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:06.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:24:06.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:24:06.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2311402469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[29779]: from='client.25921 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2311402469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:05 smithi038 ceph-mon[38113]: from='client.25921 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:06.282 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:06.384 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.427 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:06.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:06.518 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:24:06.528 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:06.556 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:06.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.696 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:06.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:06.797 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.828 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:06.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:06.930 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:06.966 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:06.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:07.067 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:07.093 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:07.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:07.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: pgmap v1001: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: from='client.25921 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:07.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[29779]: osdmap e457: 8 total, 8 up, 7 in 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: pgmap v1001: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: from='client.25921 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:07.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:06 smithi038 ceph-mon[38113]: osdmap e457: 8 total, 8 up, 7 in 2024-03-04T16:24:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.245 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:07.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:07.346 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:07.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: pgmap v1001: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: from='client.25921 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:07.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:06 smithi176 ceph-mon[33619]: osdmap e457: 8 total, 8 up, 7 in 2024-03-04T16:24:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.385 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:07.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:07.486 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.526 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:07.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:07.627 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.667 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:07.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:07.768 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:07.800 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:07.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:07.940 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:07.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:08.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:08.080 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:24:08.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:24:08.085 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:24:08.085 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:24:08.085 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:24:08.085 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:24:08.085 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:07 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:24:08.182 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:08.213 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:08.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:07 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:24:08.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:24:08.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:24:08.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:24:08.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:24:08.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:24:08.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:24:08.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:08 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:24:08.315 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:08.353 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:08.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:08.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:08.491 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:08.593 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:08.631 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:24:08.632 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [], 2024-03-04T16:24:08.632 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 0 2024-03-04T16:24:08.632 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:24:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.633 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:08.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:08.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:08.775 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:08.877 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:08.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:08.912 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:08.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:09.013 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:09.040 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:09.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.187 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:09.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:09.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:09.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:24:09.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 11.3 scrub starts 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:09.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:09 smithi176 ceph-mon[33619]: 11.3 scrub ok 2024-03-04T16:24:09.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 11.3 scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:09.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[29779]: 11.3 scrub ok 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: pgmap v1003: 148 pgs: 148 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 579 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 11.3 scrub starts 2024-03-04T16:24:09.375 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:09.376 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:09 smithi038 ceph-mon[38113]: 11.3 scrub ok 2024-03-04T16:24:09.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:09.465 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:09.500 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:24:09.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.613 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:09.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:09.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.752 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:09.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:09.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:09.863 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:09.901 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:09.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:10.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:10.044 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:10.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:10.078 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:24:10.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:10.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:10 smithi176 ceph-mon[33619]: osdmap e458: 8 total, 8 up, 7 in 2024-03-04T16:24:10.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:10 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:24:10.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:10 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:24:10.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:24:10.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:24:10.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:24:10.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:24:10.728 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:24:10.728 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[38113]: osdmap e458: 8 total, 8 up, 7 in 2024-03-04T16:24:10.728 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:24:10.728 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:24:10.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[29779]: osdmap e458: 8 total, 8 up, 7 in 2024-03-04T16:24:10.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:24:10.729 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:10 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:24:10.729 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:24:10.729 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:24:10.730 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:24:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:10.757 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:10.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:10.858 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:10.905 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:10.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:11.007 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:11.045 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:11.148 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.185 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:11.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:11.286 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:11.319 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:11.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:11 smithi176 ceph-mon[33619]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:24:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4264967956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:11 smithi176 ceph-mon[33619]: 6.f deep-scrub starts 2024-03-04T16:24:11.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:11 smithi176 ceph-mon[33619]: 6.f deep-scrub ok 2024-03-04T16:24:11.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:11.457 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:11.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[29779]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4264967956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[29779]: 6.f deep-scrub starts 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[29779]: 6.f deep-scrub ok 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[38113]: pgmap v1005: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4264967956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[38113]: 6.f deep-scrub starts 2024-03-04T16:24:11.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:11 smithi038 ceph-mon[38113]: 6.f deep-scrub ok 2024-03-04T16:24:11.560 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:11.602 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:11.704 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:11.737 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:11.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set noscrub 2024-03-04T16:24:11.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:11.900 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:11.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:12.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:12.126 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 11.8 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[29779]: 11.8 scrub ok 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 11.8 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:24:12.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:12 smithi038 ceph-mon[38113]: 11.8 scrub ok 2024-03-04T16:24:12.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:12.197 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:12.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:12.296 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:24:12.296 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:24:12.297 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 11.8 scrub starts 2024-03-04T16:24:12.297 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:24:12.297 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:24:12.297 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:12 smithi176 ceph-mon[33619]: 11.8 scrub ok 2024-03-04T16:24:12.298 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:12.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.354 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:12.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:12.455 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:12.484 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:12.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:12.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:12.616 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:12.717 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:12.747 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:12.750 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:24:12.750 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-04T16:24:12.750 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 458, 2024-03-04T16:24:12.750 INFO:teuthology.orchestra.run.smithi038.stdout: "fsid": "c453baf0-da3f-11ee-95c6-87774f69a715", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "created": "2024-03-04T15:57:15.255572+0000", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "modified": "2024-03-04T16:24:08.988695+0000", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "last_up_change": "2024-03-04T16:02:30.058990+0000", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "last_in_change": "2024-03-04T16:21:44.528259+0000", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_num": 5799936, 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_set": [ 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "pglog_hardlimit", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "purged_snapdirs", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "recovery_deletes", 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "sortbitwise" 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_version": 18, 2024-03-04T16:24:12.751 INFO:teuthology.orchestra.run.smithi038.stdout: "full_ratio": 0.94999998807907104, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_snapshot": "", 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_max": 11, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd": 8, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "require_min_compat_client": "luminous", 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "min_compat_client": "luminous", 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "require_osd_release": "squid", 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_crimson": false, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "pools": [ 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 1, 2024-03-04T16:24:12.752 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".mgr", 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:00:32.201931+0000", 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.753 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 1, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 1, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 1, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 1, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 1, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.754 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "279", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.755 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_max": 32, 2024-03-04T16:24:12.757 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_min": 1 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr": {} 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 7.8899998664855957, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 7.8899998664855957, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.37999999523162842, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 3, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 3, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.758 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 2, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "datapool", 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:41.373159+0000", 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 8193, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.759 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 3, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 3, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 3, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 3, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 3, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.760 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 2, 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 71, 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.761 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.762 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.763 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd": {} 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.6500000953674316, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.6500000953674316, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.764 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 3, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": ".rgw.root", 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:42.125847+0000", 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.765 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.766 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "69", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.767 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.768 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.769 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 2.2400000095367432, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 2.2400000095367432, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.770 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 4, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.log", 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:44.027394+0000", 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.771 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.772 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "71", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.773 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.774 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.775 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.7400000095367432, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.7400000095367432, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.776 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 5, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.control", 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:46.000786+0000", 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.777 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.778 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "73", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.779 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.780 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.781 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.7400000095367432, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.7400000095367432, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.782 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 6, 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "default.rgw.meta", 2024-03-04T16:24:12.783 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:02:48.076921+0000", 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.784 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 32, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 32, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 32, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 32, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 32, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "75", 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.785 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.786 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.787 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "options": { 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_bias": 4 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.788 INFO:teuthology.orchestra.run.smithi038.stdout: "rgw": {} 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.9900000095367432, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.9900000095367432, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.75, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.75, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.789 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "pool": 11, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": "unique_pool_4", 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "create_time": "2024-03-04T16:21:23.760037+0000", 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "flags_names": "hashpspool", 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "type": 1, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "size": 3, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": 2, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_rule": 0, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_count": 0, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_target": 0, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_barrier": 0, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-04T16:24:12.790 INFO:teuthology.orchestra.run.smithi038.stdout: "object_hash": 2, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscale_mode": "off", 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": 16, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num": 16, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_placement_num_target": 16, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_target": 16, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num_pending": 16, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_pg_merge_meta": { 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "source_pgid": "0.0", 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "ready_epoch": 0, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_started": 0, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_epoch_clean": 0, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "source_version": "0'0", 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "target_version": "0'0" 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.791 INFO:teuthology.orchestra.run.smithi038.stdout: "last_change": "404", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend": "0", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "last_force_op_resend_preluminous": "0", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "auid": 0, 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_mode": "selfmanaged", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_seq": 0, 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_epoch": 0, 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_snaps": [], 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps": "[]", 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_bytes": 0, 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "quota_max_objects": 0, 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "tiers": [], 2024-03-04T16:24:12.792 INFO:teuthology.orchestra.run.smithi038.stdout: "tier_of": -1, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "read_tier": -1, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "write_tier": -1, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_mode": "none", 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_bytes": 0, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "target_max_objects": 0, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_flush_age": 0, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "cache_min_evict_age": 0, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profile": "", 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_params": { 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "none" 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.793 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_period": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_count": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "use_gmt_hitset": true, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "min_read_recency_for_promote": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "min_write_recency_for_promote": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_grade_decay_rate": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "hit_set_search_last_n": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "grade_table": [], 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "stripe_width": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "expected_num_objects": 0, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "fast_read": false, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "options": {}, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "application_metadata": { 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "rados": {} 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.794 INFO:teuthology.orchestra.run.smithi038.stdout: "read_balance": { 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "score_type": "Fair distribution", 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "score_acting": 1.9900000095367432, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "score_stable": 1.9900000095367432, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "optimal_score": 0.87999999523162842, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_acting": 1.75, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "raw_score_stable": 1.75, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity_weighted": 0.94344520568847656, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity": 0.93999999761581421, 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "average_primary_affinity_weighted": 0.93999999761581421 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:24:12.795 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "99ecf38d-7bc9-412d-a194-f3d9eed8e244", 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7783203125, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 8, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 432, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.796 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6802", 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:24:12.797 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6804", 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.798 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6803", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1080993770 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6801/1080993770", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6802/1080993770", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6804/1080993770", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6803/1080993770", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.799 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "f8c66eb4-4966-45ce-af41-b6d01713eb51", 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 13, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 420, 2024-03-04T16:24:12.800 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6805", 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.801 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6806", 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6808", 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.802 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6807", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 2331598100 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6805/2331598100", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6806/2331598100", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6808/2331598100", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6807/2331598100", 2024-03-04T16:24:12.803 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "afb2dd7e-d27a-4536-bb80-e5724b39ffd3", 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 18, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 416, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6809", 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6810", 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.806 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6812", 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6811", 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1898914368 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6809/1898914368", 2024-03-04T16:24:12.807 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6810/1898914368", 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6812/1898914368", 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6811/1898914368", 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "13f08b82-22d5-4466-9ad4-27ffa15ed7c4", 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.808 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 0.7692413330078125, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 26, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 417, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6813", 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.809 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6814", 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6816", 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6815", 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 552035408 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.811 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.38:6813/552035408", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.38:6814/552035408", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.38:6816/552035408", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.38:6815/552035408", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "a58d697b-e1de-44cc-b791-8a8d68fd01f3", 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.812 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 32, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 358, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6800", 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:24:12.813 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6801", 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6803", 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6802", 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3997489718 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6800/3997489718", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6801/3997489718", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6803/3997489718", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6802/3997489718", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "7adb17cc-4843-401e-a7fd-b726c4a99a06", 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 42, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 432, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6804", 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:24:12.817 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6805", 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6807", 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.819 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6806", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1267814386 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6804/1267814386", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6805/1267814386", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6807/1267814386", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6806/1267814386", 2024-03-04T16:24:12.820 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "0e12d14b-c02d-4acf-9784-8680f8ee40f4", 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.821 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 50, 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 432, 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6808", 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.822 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6809", 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6811", 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:24:12.823 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6810", 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 587062426 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6808/587062426", 2024-03-04T16:24:12.824 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6809/587062426", 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6811/587062426", 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6810/587062426", 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "uuid": "1f348507-1649-4e25-b5f5-cb67c6cd045e", 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "up": 1, 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "in": 1, 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "weight": 1, 2024-03-04T16:24:12.825 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_affinity": 1, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_begin": 0, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "last_clean_end": 0, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "up_from": 59, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "up_thru": 421, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "down_at": 0, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "lost_at": 0, 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addrs": { 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6812", 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:24:12.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addrs": { 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6813", 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addrs": { 2024-03-04T16:24:12.827 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6815", 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addrs": { 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.176:6814", 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1019151537 2024-03-04T16:24:12.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "public_addr": "172.21.15.176:6812/1019151537", 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_addr": "172.21.15.176:6813/1019151537", 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_back_addr": "172.21.15.176:6815/1019151537", 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "heartbeat_front_addr": "172.21.15.176:6814/1019151537", 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "state": [ 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "exists", 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "up" 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_xinfo": [ 2024-03-04T16:24:12.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 0, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:37.446158+0000", 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 1, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.830 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:52.967064+0000", 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 2, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:31.020992+0000", 2024-03-04T16:24:12.831 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 3, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:55.805810+0000", 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.832 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 4, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 65536, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:15.820718+0000", 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.833 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 5, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:34.246205+0000", 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 6, 2024-03-04T16:24:12.834 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:24:08.274776+0000", 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "osd": 7, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "down_stamp": "0.000000", 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_probability": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "laggy_interval": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "features": 4540701547738038271, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "old_weight": 0, 2024-03-04T16:24:12.835 INFO:teuthology.orchestra.run.smithi038.stdout: "last_purged_snaps_scrub": "2024-03-04T16:23:58.380312+0000", 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: "dead_epoch": 0 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap": [ 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.b", 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: "osds": [ 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: 0, 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: 5, 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: 1 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.836 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_items": [ 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "3.b", 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 3, 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.837 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "3.1c", 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 1, 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.6", 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.838 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 1, 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.a", 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 1, 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.839 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.10", 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 3, 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 2 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "4.12", 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.840 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 1, 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.8", 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 3 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pgid": "5.13", 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "mappings": [ 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "from": 0, 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "to": 5 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_upmap_primaries": [], 2024-03-04T16:24:12.842 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_temp": [], 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "primary_temp": [], 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklist": { 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3142726897": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/452578506": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/501976220": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1665273468": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/500449596": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/4073253154": "2024-03-05T16:03:21.264346+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/175394725": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/1249306019": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:6800/2156780629": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1997350354": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:24:12.843 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2476964715": "2024-03-05T15:58:15.846725+0000", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/1178105870": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/3751720132": "2024-03-05T15:57:48.640902+0000", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "172.21.15.38:0/2774521881": "2024-03-05T15:58:15.846725+0000" 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "range_blocklist": {}, 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "erasure_code_profiles": { 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "default": { 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-failure-domain": "osd", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "k": "2", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "m": "1", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "plugin": "jerasure", 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: "technique": "reed_sol_van" 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "removed_snaps_queue": [], 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "new_removed_snaps": [], 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "new_purged_snaps": [], 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_node_flags": {}, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "device_class_flags": {}, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode": { 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_enabled": false, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_bucket_count": 0, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "degraded_stretch_mode": 0, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "recovering_stretch_mode": 0, 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: "stretch_mode_bucket": 0 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-04T16:24:12.845 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-04T16:24:12.850 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:12.893 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:12.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:12.993 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:13.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.034 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:13.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:13.103 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:13.104 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:13.130 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:24:13.135 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:13.169 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: 3.17 deep-scrub starts 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: 3.17 deep-scrub ok 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:24:13.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3284838902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: 3.17 deep-scrub starts 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: 3.17 deep-scrub ok 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:24:13.176 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3284838902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:24:13.271 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.302 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:13.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: 3.17 deep-scrub starts 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: 3.17 deep-scrub ok 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: pgmap v1006: 148 pgs: 148 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 958 op/s 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:24:13.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3284838902' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-04T16:24:13.404 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:13.434 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:13.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:13.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:13.575 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:13.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:13.726 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:13.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:13.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:24:13.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:13.871 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:13.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:13.972 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:14.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:14.103 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:14.145 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:14.234 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:14.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:14.283 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:14.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 11.7 scrub starts 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:24:14.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:14 smithi176 ceph-mon[33619]: 11.7 scrub ok 2024-03-04T16:24:14.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:14.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:24:14.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 11.7 scrub starts 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[29779]: 11.7 scrub ok 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 11.7 scrub starts 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:24:14.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:14 smithi038 ceph-mon[38113]: 11.7 scrub ok 2024-03-04T16:24:14.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:14.570 INFO:tasks.thrashosds.thrasher:j is {'epoch': 458, 'fsid': 'c453baf0-da3f-11ee-95c6-87774f69a715', 'created': '2024-03-04T15:57:15.255572+0000', 'modified': '2024-03-04T16:24:08.988695+0000', 'last_up_change': '2024-03-04T16:02:30.058990+0000', 'last_in_change': '2024-03-04T16:21:44.528259+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-04T16:00:32.201931+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '279', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-04T16:02:41.373159+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-04T16:02:42.125847+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-04T16:02:44.027394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-04T16:02:46.000786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-04T16:02:48.076921+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-04T16:21:23.760037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '404', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9434452056884766, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.9399999976158142}}], 'osds': [{'osd': 0, 'uuid': '99ecf38d-7bc9-412d-a194-f3d9eed8e244', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7783203125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6801', 'nonce': 1080993770}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6802', 'nonce': 1080993770}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6804', 'nonce': 1080993770}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6803', 'nonce': 1080993770}]}, 'public_addr': '172.21.15.38:6801/1080993770', 'cluster_addr': '172.21.15.38:6802/1080993770', 'heartbeat_back_addr': '172.21.15.38:6804/1080993770', 'heartbeat_front_addr': '172.21.15.38:6803/1080993770', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'f8c66eb4-4966-45ce-af41-b6d01713eb51', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6805', 'nonce': 2331598100}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6806', 'nonce': 2331598100}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6808', 'nonce': 2331598100}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6807', 'nonce': 2331598100}]}, 'public_addr': '172.21.15.38:6805/2331598100', 'cluster_addr': '172.21.15.38:6806/2331598100', 'heartbeat_back_addr': '172.21.15.38:6808/2331598100', 'heartbeat_front_addr': '172.21.15.38:6807/2331598100', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'afb2dd7e-d27a-4536-bb80-e5724b39ffd3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6809', 'nonce': 1898914368}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6810', 'nonce': 1898914368}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6812', 'nonce': 1898914368}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6811', 'nonce': 1898914368}]}, 'public_addr': '172.21.15.38:6809/1898914368', 'cluster_addr': '172.21.15.38:6810/1898914368', 'heartbeat_back_addr': '172.21.15.38:6812/1898914368', 'heartbeat_front_addr': '172.21.15.38:6811/1898914368', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '13f08b82-22d5-4466-9ad4-27ffa15ed7c4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7692413330078125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6813', 'nonce': 552035408}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6814', 'nonce': 552035408}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6816', 'nonce': 552035408}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.38:6815', 'nonce': 552035408}]}, 'public_addr': '172.21.15.38:6813/552035408', 'cluster_addr': '172.21.15.38:6814/552035408', 'heartbeat_back_addr': '172.21.15.38:6816/552035408', 'heartbeat_front_addr': '172.21.15.38:6815/552035408', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a58d697b-e1de-44cc-b791-8a8d68fd01f3', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 358, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6800', 'nonce': 3997489718}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6801', 'nonce': 3997489718}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6803', 'nonce': 3997489718}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6802', 'nonce': 3997489718}]}, 'public_addr': '172.21.15.176:6800/3997489718', 'cluster_addr': '172.21.15.176:6801/3997489718', 'heartbeat_back_addr': '172.21.15.176:6803/3997489718', 'heartbeat_front_addr': '172.21.15.176:6802/3997489718', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7adb17cc-4843-401e-a7fd-b726c4a99a06', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6804', 'nonce': 1267814386}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6805', 'nonce': 1267814386}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6807', 'nonce': 1267814386}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6806', 'nonce': 1267814386}]}, 'public_addr': '172.21.15.176:6804/1267814386', 'cluster_addr': '172.21.15.176:6805/1267814386', 'heartbeat_back_addr': '172.21.15.176:6807/1267814386', 'heartbeat_front_addr': '172.21.15.176:6806/1267814386', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0e12d14b-c02d-4acf-9784-8680f8ee40f4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6808', 'nonce': 587062426}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6809', 'nonce': 587062426}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6811', 'nonce': 587062426}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6810', 'nonce': 587062426}]}, 'public_addr': '172.21.15.176:6808/587062426', 'cluster_addr': '172.21.15.176:6809/587062426', 'heartbeat_back_addr': '172.21.15.176:6811/587062426', 'heartbeat_front_addr': '172.21.15.176:6810/587062426', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f348507-1649-4e25-b5f5-cb67c6cd045e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6812', 'nonce': 1019151537}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6813', 'nonce': 1019151537}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6815', 'nonce': 1019151537}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.176:6814', 'nonce': 1019151537}]}, 'public_addr': '172.21.15.176:6812/1019151537', 'cluster_addr': '172.21.15.176:6813/1019151537', 'heartbeat_back_addr': '172.21.15.176:6815/1019151537', 'heartbeat_front_addr': '172.21.15.176:6814/1019151537', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:37.446158+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:52.967064+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:31.020992+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:55.805810+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-04T16:23:15.820718+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:34.246205+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:24:08.274776+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-04T16:23:58.380312+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '4.b', 'osds': [0, 5, 1]}], 'pg_upmap_items': [{'pgid': '3.b', 'mappings': [{'from': 0, 'to': 5}, {'from': 3, 'to': 2}]}, {'pgid': '3.1c', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.10', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.8', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.13', 'mappings': [{'from': 0, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.38:0/3142726897': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/452578506': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/501976220': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/1665273468': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/500449596': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:6800/4073253154': '2024-03-05T16:03:21.264346+0000', '172.21.15.38:0/175394725': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/1249306019': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:6800/2156780629': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/1997350354': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2476964715': '2024-03-05T15:58:15.846725+0000', '172.21.15.38:0/1178105870': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/3751720132': '2024-03-05T15:57:48.640902+0000', '172.21.15.38:0/2774521881': '2024-03-05T15:58:15.846725+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-04T16:24:14.570 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:24:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:14.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:14.677 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.707 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:14.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:14.837 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:14.938 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:14.966 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:14.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:15.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:15.095 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:15.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:15.123 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:15.179 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:15.197 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:15.229 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:15.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 3.1f deep-scrub starts 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 3.1f deep-scrub ok 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: pgmap v1007: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:24:15.297 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 3.1f deep-scrub starts 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 3.1f deep-scrub ok 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: pgmap v1007: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:24:15.298 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:15 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:24:15.331 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 3.1f deep-scrub starts 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 3.1f deep-scrub ok 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: pgmap v1007: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:24:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:15 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:24:15.381 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:15.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.533 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:15.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:15.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:15.681 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:15.783 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:15.810 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:15.912 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:15.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:15.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:16.043 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:16.070 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:16.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:16.211 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:16.309 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:16.310 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:16.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:16.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.347 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:16.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:16.448 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:16.478 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:16.580 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:16 smithi176 ceph-mon[33619]: 5.1f scrub starts 2024-03-04T16:24:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:16 smithi176 ceph-mon[33619]: 5.1f scrub ok 2024-03-04T16:24:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:16 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:24:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:16 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:24:16.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:16 smithi176 ceph-mon[33619]: pgmap v1008: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:16.609 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[29779]: 5.1f scrub starts 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[29779]: 5.1f scrub ok 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[29779]: pgmap v1008: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[38113]: 5.1f scrub starts 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[38113]: 5.1f scrub ok 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:24:16.655 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:16 smithi038 ceph-mon[38113]: pgmap v1008: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-04T16:24:16.711 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:16.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:16.902 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:16.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:17.004 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:17.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:24:17.039 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:17.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:17.141 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.177 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:17.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:17.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:17.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:17.364 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:17.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:17.455 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:17.489 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is set 2024-03-04T16:24:17.557 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:17.587 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:17.689 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:17.720 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: 4.10 deep-scrub starts 2024-03-04T16:24:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: 4.10 deep-scrub ok 2024-03-04T16:24:17.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: 4.10 deep-scrub starts 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: 4.10 deep-scrub ok 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:24:17.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:24:17.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:17 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:24:17.822 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:17.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: 4.10 deep-scrub starts 2024-03-04T16:24:17.842 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: 4.10 deep-scrub ok 2024-03-04T16:24:17.842 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:24:17.842 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:24:17.842 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-04T16:24:17.842 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:17 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:24:17.851 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:17.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:17.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:17.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:17.999 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:18.101 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.152 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:18.253 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:18.399 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:18.429 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:18.531 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:18.576 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:18.678 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:18.715 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:18.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:24:18.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:24:18.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:18.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:24:18.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: osdmap e459: 8 total, 8 up, 7 in 2024-03-04T16:24:18.841 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:18 smithi176 ceph-mon[33619]: pgmap v1010: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:24:18.848 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:18.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:18.988 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:18.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:18.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:24:18.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:24:18.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:24:18.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: osdmap e459: 8 total, 8 up, 7 in 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[29779]: pgmap v1010: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:24:18.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3428150041' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: osdmap e459: 8 total, 8 up, 7 in 2024-03-04T16:24:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:18 smithi038 ceph-mon[38113]: pgmap v1010: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-04T16:24:19.090 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:19.108 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:19.108 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:19.127 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:19.230 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:19.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:19.370 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.405 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:19.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:19.466 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:24:19.469 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:24:19.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:19.534 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:19.635 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:19.663 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:19.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:19 smithi038 ceph-mon[29779]: from='client.36014 v1:172.21.15.38:0/2614146528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:24:19.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:19 smithi038 ceph-mon[38113]: from='client.36014 v1:172.21.15.38:0/2614146528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:24:19.765 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:19.794 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:19.832 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:19 smithi176 ceph-mon[33619]: from='client.36014 v1:172.21.15.38:0/2614146528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:24:19.872 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:24:19.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:19.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:19.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:20.043 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:20.084 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:20.186 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:20.220 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:20.247 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:20.323 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.368 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:20.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:20.469 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:20.499 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:20.602 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:20.646 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:20.669 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:20 smithi176 ceph-mon[33619]: pgmap v1011: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:20.749 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.784 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:20.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:20.876 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:20 smithi038 ceph-mon[29779]: pgmap v1011: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:20.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:20 smithi038 ceph-mon[38113]: pgmap v1011: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:20.885 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:20.924 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:20.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:20.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:21.027 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:21.056 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:21.158 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:21.193 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:21.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.346 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:21.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:21.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.506 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:21.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:21.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:21.640 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:21.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:21.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:21.771 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:21.873 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:21.920 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:21.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:22.021 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:22.031 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:24:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a[63952]: logger=cleanup t=2024-03-04T16:24:21.737595693Z level=info msg="Completed cleanup jobs" duration=2.20035ms 2024-03-04T16:24:22.060 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:22.104 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:22.105 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:22.162 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:22.192 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:22.295 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.336 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:22.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:22.349 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1010,"stamp":"2024-03-04T16:24:17.764539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2132416886,"num_objects":32728,"num_object_clones":0,"num_object_copies":98184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32728,"num_whiteouts":0,"num_read":3048,"num_read_kb":2894,"num_write":182931,"num_write_kb":6859166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8805,"num_bytes_recovered":548685156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77628,"ondisk_log_size":77628,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12640532,"kb_used_data":6411868,"kb_used_omap":10,"kb_used_meta":6228469,"kb_avail":643575532,"statfs":{"total":671965249536,"available":659021344768,"internally_reserved":0,"allocated":6565752832,"data_stored":6563055002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":6377952365},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-592576512,"num_objects":-9042,"num_object_clones":0,"num_object_copies":-27126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9042,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9042,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.765789"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":207,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009066+0000","last_change":"2024-03-04T16:21:44.712801+0000","last_active":"2024-03-04T16:24:09.009066+0000","last_peered":"2024-03-04T16:24:09.009066+0000","last_clean":"2024-03-04T16:24:09.009066+0000","last_became_active":"2024-03-04T16:20:07.476108+0000","last_became_peered":"2024-03-04T16:20:07.476108+0000","last_unstale":"2024-03-04T16:24:09.009066+0000","last_undegraded":"2024-03-04T16:24:09.009066+0000","last_fullsized":"2024-03-04T16:24:09.009066+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.712721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:22.868660+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.712721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":855,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012461+0000","last_change":"2024-03-04T16:24:01.547883+0000","last_active":"2024-03-04T16:24:09.012461+0000","last_peered":"2024-03-04T16:24:09.012461+0000","last_clean":"2024-03-04T16:24:09.012461+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:24:09.012461+0000","last_undegraded":"2024-03-04T16:24:09.012461+0000","last_fullsized":"2024-03-04T16:24:09.012461+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:01.547751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:23.794378+0000","last_clean_scrub_stamp":"2024-03-04T16:24:01.547751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:16.367247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":726,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:13.597806+0000","last_change":"2024-03-04T16:24:13.597844+0000","last_active":"2024-03-04T16:24:13.597806+0000","last_peered":"2024-03-04T16:24:13.597806+0000","last_clean":"2024-03-04T16:24:13.597806+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:24:13.597806+0000","last_undegraded":"2024-03-04T16:24:13.597806+0000","last_fullsized":"2024-03-04T16:24:13.597806+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:13.597504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:13.597504+0000","last_clean_scrub_stamp":"2024-03-04T16:24:13.597504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:23.584414+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":139,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490331+0000","last_change":"2024-03-04T16:24:13.064829+0000","last_active":"2024-03-04T16:24:17.490331+0000","last_peered":"2024-03-04T16:24:17.490331+0000","last_clean":"2024-03-04T16:24:17.490331+0000","last_became_active":"2024-03-04T16:21:45.423494+0000","last_became_peered":"2024-03-04T16:21:45.423494+0000","last_unstale":"2024-03-04T16:24:17.490331+0000","last_undegraded":"2024-03-04T16:24:17.490331+0000","last_fullsized":"2024-03-04T16:24:17.490331+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:24:13.064640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:24:13.064640+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:16.045190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":855,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.026412+0000","last_change":"2024-03-04T16:23:59.316470+0000","last_active":"2024-03-04T16:24:09.026412+0000","last_peered":"2024-03-04T16:24:09.026412+0000","last_clean":"2024-03-04T16:24:09.026412+0000","last_became_active":"2024-03-04T16:21:45.427819+0000","last_became_peered":"2024-03-04T16:21:45.427819+0000","last_unstale":"2024-03-04T16:24:09.026412+0000","last_undegraded":"2024-03-04T16:24:09.026412+0000","last_fullsized":"2024-03-04T16:24:09.026412+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:59.316337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:23:59.316337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:15.601316+0000","scrub_duration":264,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490494+0000","last_change":"2024-03-04T16:22:56.684863+0000","last_active":"2024-03-04T16:24:17.490494+0000","last_peered":"2024-03-04T16:24:17.490494+0000","last_clean":"2024-03-04T16:24:17.490494+0000","last_became_active":"2024-03-04T16:21:45.422482+0000","last_became_peered":"2024-03-04T16:21:45.422482+0000","last_unstale":"2024-03-04T16:24:17.490494+0000","last_undegraded":"2024-03-04T16:24:17.490494+0000","last_fullsized":"2024-03-04T16:24:17.490494+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.684724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.684724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.493825+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":873,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.022928+0000","last_change":"2024-03-04T16:23:08.925901+0000","last_active":"2024-03-04T16:24:09.022928+0000","last_peered":"2024-03-04T16:24:09.022928+0000","last_clean":"2024-03-04T16:24:09.022928+0000","last_became_active":"2024-03-04T16:21:45.422756+0000","last_became_peered":"2024-03-04T16:21:45.422756+0000","last_unstale":"2024-03-04T16:24:09.022928+0000","last_undegraded":"2024-03-04T16:24:09.022928+0000","last_fullsized":"2024-03-04T16:24:09.022928+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.925783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.299467+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'39","reported_seq":850,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.017749+0000","last_change":"2024-03-04T16:22:47.640816+0000","last_active":"2024-03-04T16:24:09.017749+0000","last_peered":"2024-03-04T16:24:09.017749+0000","last_clean":"2024-03-04T16:24:09.017749+0000","last_became_active":"2024-03-04T16:21:25.444226+0000","last_became_peered":"2024-03-04T16:21:25.444226+0000","last_unstale":"2024-03-04T16:24:09.017749+0000","last_undegraded":"2024-03-04T16:24:09.017749+0000","last_fullsized":"2024-03-04T16:24:09.017749+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'27","last_scrub_stamp":"2024-03-04T16:22:47.640638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.640638+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":707,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.027698+0000","last_change":"2024-03-04T16:22:50.786398+0000","last_active":"2024-03-04T16:24:09.027698+0000","last_peered":"2024-03-04T16:24:09.027698+0000","last_clean":"2024-03-04T16:24:09.027698+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:24:09.027698+0000","last_undegraded":"2024-03-04T16:24:09.027698+0000","last_fullsized":"2024-03-04T16:24:09.027698+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.786274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.786274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":846,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009216+0000","last_change":"2024-03-04T16:22:06.888682+0000","last_active":"2024-03-04T16:24:09.009216+0000","last_peered":"2024-03-04T16:24:09.009216+0000","last_clean":"2024-03-04T16:24:09.009216+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:24:09.009216+0000","last_undegraded":"2024-03-04T16:24:09.009216+0000","last_fullsized":"2024-03-04T16:24:09.009216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.888592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":880,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009902+0000","last_change":"2024-03-04T16:21:45.443994+0000","last_active":"2024-03-04T16:24:09.009902+0000","last_peered":"2024-03-04T16:24:09.009902+0000","last_clean":"2024-03-04T16:24:09.009902+0000","last_became_active":"2024-03-04T16:21:45.443239+0000","last_became_peered":"2024-03-04T16:21:45.443239+0000","last_unstale":"2024-03-04T16:24:09.009902+0000","last_undegraded":"2024-03-04T16:24:09.009902+0000","last_fullsized":"2024-03-04T16:24:09.009902+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:36.877837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:36.877837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'25","reported_seq":680,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.112368+0000","last_change":"2024-03-04T16:24:09.112433+0000","last_active":"2024-03-04T16:24:09.112368+0000","last_peered":"2024-03-04T16:24:09.112368+0000","last_clean":"2024-03-04T16:24:09.112368+0000","last_became_active":"2024-03-04T16:20:07.491874+0000","last_became_peered":"2024-03-04T16:20:07.491874+0000","last_unstale":"2024-03-04T16:24:09.112368+0000","last_undegraded":"2024-03-04T16:24:09.112368+0000","last_fullsized":"2024-03-04T16:24:09.112368+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:24:09.112284+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:24:09.112284+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:33.126242+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":858,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.008851+0000","last_change":"2024-03-04T16:24:05.063871+0000","last_active":"2024-03-04T16:24:09.008851+0000","last_peered":"2024-03-04T16:24:09.008851+0000","last_clean":"2024-03-04T16:24:09.008851+0000","last_became_active":"2024-03-04T16:02:49.791185+0000","last_became_peered":"2024-03-04T16:02:49.791185+0000","last_unstale":"2024-03-04T16:24:09.008851+0000","last_undegraded":"2024-03-04T16:24:09.008851+0000","last_fullsized":"2024-03-04T16:24:09.008851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:05.063725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:24:05.063725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:34.849972+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996585+0000","last_change":"2024-03-04T16:22:46.297643+0000","last_active":"2024-03-04T16:24:08.996585+0000","last_peered":"2024-03-04T16:24:08.996585+0000","last_clean":"2024-03-04T16:24:08.996585+0000","last_became_active":"2024-03-04T16:20:07.494164+0000","last_became_peered":"2024-03-04T16:20:07.494164+0000","last_unstale":"2024-03-04T16:24:08.996585+0000","last_undegraded":"2024-03-04T16:24:08.996585+0000","last_fullsized":"2024-03-04T16:24:08.996585+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.297535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.297535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:12.975907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":880,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009603+0000","last_change":"2024-03-04T16:24:07.935484+0000","last_active":"2024-03-04T16:24:09.009603+0000","last_peered":"2024-03-04T16:24:09.009603+0000","last_clean":"2024-03-04T16:24:09.009603+0000","last_became_active":"2024-03-04T16:21:45.422460+0000","last_became_peered":"2024-03-04T16:21:45.422460+0000","last_unstale":"2024-03-04T16:24:09.009603+0000","last_undegraded":"2024-03-04T16:24:09.009603+0000","last_fullsized":"2024-03-04T16:24:09.009603+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:07.935406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:24:07.935406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:14.797764+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"441'11","reported_seq":640,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.026264+0000","last_change":"2024-03-04T16:23:00.985610+0000","last_active":"2024-03-04T16:24:09.026264+0000","last_peered":"2024-03-04T16:24:09.026264+0000","last_clean":"2024-03-04T16:24:09.026264+0000","last_became_active":"2024-03-04T16:20:07.482406+0000","last_became_peered":"2024-03-04T16:20:07.482406+0000","last_unstale":"2024-03-04T16:24:09.026264+0000","last_undegraded":"2024-03-04T16:24:09.026264+0000","last_fullsized":"2024-03-04T16:24:09.026264+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'11","last_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_deep_scrub":"441'11","last_deep_scrub_stamp":"2024-03-04T16:23:00.985229+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.985229+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.484493+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490905+0000","last_change":"2024-03-04T16:22:59.694873+0000","last_active":"2024-03-04T16:24:17.490905+0000","last_peered":"2024-03-04T16:24:17.490905+0000","last_clean":"2024-03-04T16:24:17.490905+0000","last_became_active":"2024-03-04T16:21:45.425723+0000","last_became_peered":"2024-03-04T16:21:45.425723+0000","last_unstale":"2024-03-04T16:24:17.490905+0000","last_undegraded":"2024-03-04T16:24:17.490905+0000","last_fullsized":"2024-03-04T16:24:17.490905+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:59.694757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:59.694757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:19.218070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.016396+0000","last_change":"2024-03-04T16:23:55.504879+0000","last_active":"2024-03-04T16:24:09.016396+0000","last_peered":"2024-03-04T16:24:09.016396+0000","last_clean":"2024-03-04T16:24:09.016396+0000","last_became_active":"2024-03-04T16:20:07.497650+0000","last_became_peered":"2024-03-04T16:20:07.497650+0000","last_unstale":"2024-03-04T16:24:09.016396+0000","last_undegraded":"2024-03-04T16:24:09.016396+0000","last_fullsized":"2024-03-04T16:24:09.016396+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:55.504423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:55.504423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:20.365153+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":705,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.031504+0000","last_change":"2024-03-04T16:22:45.654818+0000","last_active":"2024-03-04T16:24:09.031504+0000","last_peered":"2024-03-04T16:24:09.031504+0000","last_clean":"2024-03-04T16:24:09.031504+0000","last_became_active":"2024-03-04T16:20:07.498539+0000","last_became_peered":"2024-03-04T16:20:07.498539+0000","last_unstale":"2024-03-04T16:24:09.031504+0000","last_undegraded":"2024-03-04T16:24:09.031504+0000","last_fullsized":"2024-03-04T16:24:09.031504+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.654354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.654354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.175429+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":303,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012042+0000","last_change":"2024-03-04T16:24:05.122244+0000","last_active":"2024-03-04T16:24:09.012042+0000","last_peered":"2024-03-04T16:24:09.012042+0000","last_clean":"2024-03-04T16:24:09.012042+0000","last_became_active":"2024-03-04T16:19:04.771891+0000","last_became_peered":"2024-03-04T16:19:04.771891+0000","last_unstale":"2024-03-04T16:24:09.012042+0000","last_undegraded":"2024-03-04T16:24:09.012042+0000","last_fullsized":"2024-03-04T16:24:09.012042+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:24:05.121945+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:24:05.121945+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:09.754030+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490762+0000","last_change":"2024-03-04T16:22:43.806087+0000","last_active":"2024-03-04T16:24:17.490762+0000","last_peered":"2024-03-04T16:24:17.490762+0000","last_clean":"2024-03-04T16:24:17.490762+0000","last_became_active":"2024-03-04T16:21:45.420331+0000","last_became_peered":"2024-03-04T16:21:45.420331+0000","last_unstale":"2024-03-04T16:24:17.490762+0000","last_undegraded":"2024-03-04T16:24:17.490762+0000","last_fullsized":"2024-03-04T16:24:17.490762+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.805945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.805945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":815,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.022859+0000","last_change":"2024-03-04T16:24:03.145035+0000","last_active":"2024-03-04T16:24:09.022859+0000","last_peered":"2024-03-04T16:24:09.022859+0000","last_clean":"2024-03-04T16:24:09.022859+0000","last_became_active":"2024-03-04T16:20:07.474386+0000","last_became_peered":"2024-03-04T16:20:07.474386+0000","last_unstale":"2024-03-04T16:24:09.022859+0000","last_undegraded":"2024-03-04T16:24:09.022859+0000","last_fullsized":"2024-03-04T16:24:09.022859+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:03.144918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:03.144918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:24.671021+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":823,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.016741+0000","last_change":"2024-03-04T16:22:56.344093+0000","last_active":"2024-03-04T16:24:09.016741+0000","last_peered":"2024-03-04T16:24:09.016741+0000","last_clean":"2024-03-04T16:24:09.016741+0000","last_became_active":"2024-03-04T16:20:07.503831+0000","last_became_peered":"2024-03-04T16:20:07.503831+0000","last_unstale":"2024-03-04T16:24:09.016741+0000","last_undegraded":"2024-03-04T16:24:09.016741+0000","last_fullsized":"2024-03-04T16:24:09.016741+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:56.343792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.343792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":577,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.016176+0000","last_change":"2024-03-04T16:23:57.549123+0000","last_active":"2024-03-04T16:24:09.016176+0000","last_peered":"2024-03-04T16:24:09.016176+0000","last_clean":"2024-03-04T16:24:09.016176+0000","last_became_active":"2024-03-04T16:21:45.441225+0000","last_became_peered":"2024-03-04T16:21:45.441225+0000","last_unstale":"2024-03-04T16:24:09.016176+0000","last_undegraded":"2024-03-04T16:24:09.016176+0000","last_fullsized":"2024-03-04T16:24:09.016176+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:23:57.548982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:57.548982+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:14.569709+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":873,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997719+0000","last_change":"2024-03-04T16:22:45.271732+0000","last_active":"2024-03-04T16:24:08.997719+0000","last_peered":"2024-03-04T16:24:08.997719+0000","last_clean":"2024-03-04T16:24:08.997719+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:24:08.997719+0000","last_undegraded":"2024-03-04T16:24:08.997719+0000","last_fullsized":"2024-03-04T16:24:08.997719+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:22:45.271595+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:04:17.870988+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.271595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"412'13","reported_seq":154,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490483+0000","last_change":"2024-03-04T16:24:15.054016+0000","last_active":"2024-03-04T16:24:17.490483+0000","last_peered":"2024-03-04T16:24:17.490483+0000","last_clean":"2024-03-04T16:24:17.490483+0000","last_became_active":"2024-03-04T16:21:45.416857+0000","last_became_peered":"2024-03-04T16:21:45.416857+0000","last_unstale":"2024-03-04T16:24:17.490483+0000","last_undegraded":"2024-03-04T16:24:17.490483+0000","last_fullsized":"2024-03-04T16:24:17.490483+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'13","last_scrub_stamp":"2024-03-04T16:24:15.053924+0000","last_deep_scrub":"195'10","last_deep_scrub_stamp":"2024-03-04T16:17:08.205209+0000","last_clean_scrub_stamp":"2024-03-04T16:24:15.053924+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:42.389557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":884,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012713+0000","last_change":"2024-03-04T16:22:43.057543+0000","last_active":"2024-03-04T16:24:09.012713+0000","last_peered":"2024-03-04T16:24:09.012713+0000","last_clean":"2024-03-04T16:24:09.012713+0000","last_became_active":"2024-03-04T16:21:45.437760+0000","last_became_peered":"2024-03-04T16:21:45.437760+0000","last_unstale":"2024-03-04T16:24:09.012713+0000","last_undegraded":"2024-03-04T16:24:09.012713+0000","last_fullsized":"2024-03-04T16:24:09.012713+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:43.057419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.057419+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":751,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.032088+0000","last_change":"2024-03-04T16:23:02.578786+0000","last_active":"2024-03-04T16:24:09.032088+0000","last_peered":"2024-03-04T16:24:09.032088+0000","last_clean":"2024-03-04T16:24:09.032088+0000","last_became_active":"2024-03-04T16:07:36.236790+0000","last_became_peered":"2024-03-04T16:07:36.236790+0000","last_unstale":"2024-03-04T16:24:09.032088+0000","last_undegraded":"2024-03-04T16:24:09.032088+0000","last_fullsized":"2024-03-04T16:24:09.032088+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:23:02.578648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.578648+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":857,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.016544+0000","last_change":"2024-03-04T16:23:06.666597+0000","last_active":"2024-03-04T16:24:09.016544+0000","last_peered":"2024-03-04T16:24:09.016544+0000","last_clean":"2024-03-04T16:24:09.016544+0000","last_became_active":"2024-03-04T16:20:07.477199+0000","last_became_peered":"2024-03-04T16:20:07.477199+0000","last_unstale":"2024-03-04T16:24:09.016544+0000","last_undegraded":"2024-03-04T16:24:09.016544+0000","last_fullsized":"2024-03-04T16:24:09.016544+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:06.666475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:08.786009+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.666475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:33.618544+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":384,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.031069+0000","last_change":"2024-03-04T16:22:57.654317+0000","last_active":"2024-03-04T16:24:09.031069+0000","last_peered":"2024-03-04T16:24:09.031069+0000","last_clean":"2024-03-04T16:24:09.031069+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:24:09.031069+0000","last_undegraded":"2024-03-04T16:24:09.031069+0000","last_fullsized":"2024-03-04T16:24:09.031069+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:57.653816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.653816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.989031+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":867,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.023153+0000","last_change":"2024-03-04T16:24:01.183149+0000","last_active":"2024-03-04T16:24:09.023153+0000","last_peered":"2024-03-04T16:24:09.023153+0000","last_clean":"2024-03-04T16:24:09.023153+0000","last_became_active":"2024-03-04T16:07:36.210624+0000","last_became_peered":"2024-03-04T16:07:36.210624+0000","last_unstale":"2024-03-04T16:24:09.023153+0000","last_undegraded":"2024-03-04T16:24:09.023153+0000","last_fullsized":"2024-03-04T16:24:09.023153+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:01.182732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:24:01.182732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:15.281419+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'27","reported_seq":144,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490660+0000","last_change":"2024-03-04T16:22:42.533910+0000","last_active":"2024-03-04T16:24:17.490660+0000","last_peered":"2024-03-04T16:24:17.490660+0000","last_clean":"2024-03-04T16:24:17.490660+0000","last_became_active":"2024-03-04T16:21:45.434840+0000","last_became_peered":"2024-03-04T16:21:45.434840+0000","last_unstale":"2024-03-04T16:24:17.490660+0000","last_undegraded":"2024-03-04T16:24:17.490660+0000","last_fullsized":"2024-03-04T16:24:17.490660+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'19","last_scrub_stamp":"2024-03-04T16:22:42.533824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.533824+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":878,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996673+0000","last_change":"2024-03-04T16:22:09.161333+0000","last_active":"2024-03-04T16:24:08.996673+0000","last_peered":"2024-03-04T16:24:08.996673+0000","last_clean":"2024-03-04T16:24:08.996673+0000","last_became_active":"2024-03-04T16:20:07.477661+0000","last_became_peered":"2024-03-04T16:20:07.477661+0000","last_unstale":"2024-03-04T16:24:08.996673+0000","last_undegraded":"2024-03-04T16:24:08.996673+0000","last_fullsized":"2024-03-04T16:24:08.996673+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.161217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.161217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.492529+0000","last_change":"2024-03-04T16:23:08.842921+0000","last_active":"2024-03-04T16:24:17.492529+0000","last_peered":"2024-03-04T16:24:17.492529+0000","last_clean":"2024-03-04T16:24:17.492529+0000","last_became_active":"2024-03-04T16:21:45.414027+0000","last_became_peered":"2024-03-04T16:21:45.414027+0000","last_unstale":"2024-03-04T16:24:17.492529+0000","last_undegraded":"2024-03-04T16:24:17.492529+0000","last_fullsized":"2024-03-04T16:24:17.492529+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:08.842842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.842842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:21.737638+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":707,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:11.629133+0000","last_change":"2024-03-04T16:24:11.629169+0000","last_active":"2024-03-04T16:24:11.629133+0000","last_peered":"2024-03-04T16:24:11.629133+0000","last_clean":"2024-03-04T16:24:11.629133+0000","last_became_active":"2024-03-04T16:07:36.220887+0000","last_became_peered":"2024-03-04T16:07:36.220887+0000","last_unstale":"2024-03-04T16:24:11.629133+0000","last_undegraded":"2024-03-04T16:24:11.629133+0000","last_fullsized":"2024-03-04T16:24:11.629133+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:11.629031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:11.629031+0000","last_clean_scrub_stamp":"2024-03-04T16:24:11.629031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:30.000528+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"441'12","reported_seq":166,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.017668+0000","last_change":"2024-03-04T16:22:50.599109+0000","last_active":"2024-03-04T16:24:09.017668+0000","last_peered":"2024-03-04T16:24:09.017668+0000","last_clean":"2024-03-04T16:24:09.017668+0000","last_became_active":"2024-03-04T16:21:45.423994+0000","last_became_peered":"2024-03-04T16:21:45.423994+0000","last_unstale":"2024-03-04T16:24:09.017668+0000","last_undegraded":"2024-03-04T16:24:09.017668+0000","last_fullsized":"2024-03-04T16:24:09.017668+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'8","last_scrub_stamp":"2024-03-04T16:22:50.598967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.598967+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":866,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.026586+0000","last_change":"2024-03-04T16:23:03.173401+0000","last_active":"2024-03-04T16:24:09.026586+0000","last_peered":"2024-03-04T16:24:09.026586+0000","last_clean":"2024-03-04T16:24:09.026586+0000","last_became_active":"2024-03-04T16:21:45.429844+0000","last_became_peered":"2024-03-04T16:21:45.429844+0000","last_unstale":"2024-03-04T16:24:09.026586+0000","last_undegraded":"2024-03-04T16:24:09.026586+0000","last_fullsized":"2024-03-04T16:24:09.026586+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.173174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.173174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.851084+0000","scrub_duration":244,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":874,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996481+0000","last_change":"2024-03-04T16:22:54.333690+0000","last_active":"2024-03-04T16:24:08.996481+0000","last_peered":"2024-03-04T16:24:08.996481+0000","last_clean":"2024-03-04T16:24:08.996481+0000","last_became_active":"2024-03-04T16:21:45.415087+0000","last_became_peered":"2024-03-04T16:21:45.415087+0000","last_unstale":"2024-03-04T16:24:08.996481+0000","last_undegraded":"2024-03-04T16:24:08.996481+0000","last_fullsized":"2024-03-04T16:24:08.996481+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.333535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:49.987677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.333535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":862,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010150+0000","last_change":"2024-03-04T16:22:41.028799+0000","last_active":"2024-03-04T16:24:09.010150+0000","last_peered":"2024-03-04T16:24:09.010150+0000","last_clean":"2024-03-04T16:24:09.010150+0000","last_became_active":"2024-03-04T16:02:43.713270+0000","last_became_peered":"2024-03-04T16:02:43.713270+0000","last_unstale":"2024-03-04T16:24:09.010150+0000","last_undegraded":"2024-03-04T16:24:09.010150+0000","last_fullsized":"2024-03-04T16:24:09.010150+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.028680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.028680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'27","reported_seq":923,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.027324+0000","last_change":"2024-03-04T16:22:55.927787+0000","last_active":"2024-03-04T16:24:09.027324+0000","last_peered":"2024-03-04T16:24:09.027324+0000","last_clean":"2024-03-04T16:24:09.027324+0000","last_became_active":"2024-03-04T16:21:45.428057+0000","last_became_peered":"2024-03-04T16:21:45.428057+0000","last_unstale":"2024-03-04T16:24:09.027324+0000","last_undegraded":"2024-03-04T16:24:09.027324+0000","last_fullsized":"2024-03-04T16:24:09.027324+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:22:55.927235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:55.927235+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":700,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.027904+0000","last_change":"2024-03-04T16:22:46.734503+0000","last_active":"2024-03-04T16:24:09.027904+0000","last_peered":"2024-03-04T16:24:09.027904+0000","last_clean":"2024-03-04T16:24:09.027904+0000","last_became_active":"2024-03-04T16:07:36.224960+0000","last_became_peered":"2024-03-04T16:07:36.224960+0000","last_unstale":"2024-03-04T16:24:09.027904+0000","last_undegraded":"2024-03-04T16:24:09.027904+0000","last_fullsized":"2024-03-04T16:24:09.027904+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.734392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:10.175912+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.734392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:10.402153+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":497,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009129+0000","last_change":"2024-03-04T16:24:05.953409+0000","last_active":"2024-03-04T16:24:09.009129+0000","last_peered":"2024-03-04T16:24:09.009129+0000","last_clean":"2024-03-04T16:24:09.009129+0000","last_became_active":"2024-03-04T16:21:25.449034+0000","last_became_peered":"2024-03-04T16:21:25.449034+0000","last_unstale":"2024-03-04T16:24:09.009129+0000","last_undegraded":"2024-03-04T16:24:09.009129+0000","last_fullsized":"2024-03-04T16:24:09.009129+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:05.953233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:05.953233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:22.303177+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996877+0000","last_change":"2024-03-04T16:21:58.944553+0000","last_active":"2024-03-04T16:24:08.996877+0000","last_peered":"2024-03-04T16:24:08.996877+0000","last_clean":"2024-03-04T16:24:08.996877+0000","last_became_active":"2024-03-04T16:20:07.498410+0000","last_became_peered":"2024-03-04T16:20:07.498410+0000","last_unstale":"2024-03-04T16:24:08.996877+0000","last_undegraded":"2024-03-04T16:24:08.996877+0000","last_fullsized":"2024-03-04T16:24:08.996877+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:58.944477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:58.944477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'25","reported_seq":96,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.008967+0000","last_change":"2024-03-04T16:23:52.014266+0000","last_active":"2024-03-04T16:24:09.008967+0000","last_peered":"2024-03-04T16:24:09.008967+0000","last_clean":"2024-03-04T16:24:09.008967+0000","last_became_active":"2024-03-04T16:22:24.825469+0000","last_became_peered":"2024-03-04T16:22:24.825469+0000","last_unstale":"2024-03-04T16:24:09.008967+0000","last_undegraded":"2024-03-04T16:24:09.008967+0000","last_fullsized":"2024-03-04T16:24:09.008967+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:23:52.014128+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:23:52.014128+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:14.597749+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":291,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.035792+0000","last_change":"2024-03-04T16:22:44.719886+0000","last_active":"2024-03-04T16:24:09.035792+0000","last_peered":"2024-03-04T16:24:09.035792+0000","last_clean":"2024-03-04T16:24:09.035792+0000","last_became_active":"2024-03-04T16:21:25.446654+0000","last_became_peered":"2024-03-04T16:21:25.446654+0000","last_unstale":"2024-03-04T16:24:09.035792+0000","last_undegraded":"2024-03-04T16:24:09.035792+0000","last_fullsized":"2024-03-04T16:24:09.035792+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.719762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:13:13.530037+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.719762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":856,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.020128+0000","last_change":"2024-03-04T16:22:47.060545+0000","last_active":"2024-03-04T16:24:09.020128+0000","last_peered":"2024-03-04T16:24:09.020128+0000","last_clean":"2024-03-04T16:24:09.020128+0000","last_became_active":"2024-03-04T16:02:47.769910+0000","last_became_peered":"2024-03-04T16:02:47.769910+0000","last_unstale":"2024-03-04T16:24:09.020128+0000","last_undegraded":"2024-03-04T16:24:09.020128+0000","last_fullsized":"2024-03-04T16:24:09.020128+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.060423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.060423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":821,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997631+0000","last_change":"2024-03-04T16:21:45.416172+0000","last_active":"2024-03-04T16:24:08.997631+0000","last_peered":"2024-03-04T16:24:08.997631+0000","last_clean":"2024-03-04T16:24:08.997631+0000","last_became_active":"2024-03-04T16:21:45.415525+0000","last_became_peered":"2024-03-04T16:21:45.415525+0000","last_unstale":"2024-03-04T16:24:08.997631+0000","last_undegraded":"2024-03-04T16:24:08.997631+0000","last_fullsized":"2024-03-04T16:24:08.997631+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:37.800150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:37.800150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":144,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490694+0000","last_change":"2024-03-04T16:22:52.714265+0000","last_active":"2024-03-04T16:24:17.490694+0000","last_peered":"2024-03-04T16:24:17.490694+0000","last_clean":"2024-03-04T16:24:17.490694+0000","last_became_active":"2024-03-04T16:21:45.437462+0000","last_became_peered":"2024-03-04T16:21:45.437462+0000","last_unstale":"2024-03-04T16:24:17.490694+0000","last_undegraded":"2024-03-04T16:24:17.490694+0000","last_fullsized":"2024-03-04T16:24:17.490694+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:22:52.714182+0000","last_deep_scrub":"316'24","last_deep_scrub_stamp":"2024-03-04T16:18:04.512502+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.714182+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":733,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997036+0000","last_change":"2024-03-04T16:23:11.346983+0000","last_active":"2024-03-04T16:24:08.997036+0000","last_peered":"2024-03-04T16:24:08.997036+0000","last_clean":"2024-03-04T16:24:08.997036+0000","last_became_active":"2024-03-04T16:07:36.235979+0000","last_became_peered":"2024-03-04T16:07:36.235979+0000","last_unstale":"2024-03-04T16:24:08.997036+0000","last_undegraded":"2024-03-04T16:24:08.997036+0000","last_fullsized":"2024-03-04T16:24:08.997036+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:11.346910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:11.346910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.225627+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"141'10","reported_seq":1134,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010029+0000","last_change":"2024-03-04T16:22:04.841240+0000","last_active":"2024-03-04T16:24:09.010029+0000","last_peered":"2024-03-04T16:24:09.010029+0000","last_clean":"2024-03-04T16:24:09.010029+0000","last_became_active":"2024-03-04T16:07:36.224083+0000","last_became_peered":"2024-03-04T16:07:36.224083+0000","last_unstale":"2024-03-04T16:24:09.010029+0000","last_undegraded":"2024-03-04T16:24:09.010029+0000","last_fullsized":"2024-03-04T16:24:09.010029+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'10","last_scrub_stamp":"2024-03-04T16:22:04.841055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.841055+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"68'1","reported_seq":928,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996776+0000","last_change":"2024-03-04T16:23:03.314418+0000","last_active":"2024-03-04T16:24:08.996776+0000","last_peered":"2024-03-04T16:24:08.996776+0000","last_clean":"2024-03-04T16:24:08.996776+0000","last_became_active":"2024-03-04T16:21:45.413361+0000","last_became_peered":"2024-03-04T16:21:45.413361+0000","last_unstale":"2024-03-04T16:24:08.996776+0000","last_undegraded":"2024-03-04T16:24:08.996776+0000","last_fullsized":"2024-03-04T16:24:08.996776+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:23:03.314294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.314294+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:26.292076+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'30","reported_seq":591,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012410+0000","last_change":"2024-03-04T16:23:56.504640+0000","last_active":"2024-03-04T16:24:09.012410+0000","last_peered":"2024-03-04T16:24:09.012410+0000","last_clean":"2024-03-04T16:24:09.012410+0000","last_became_active":"2024-03-04T16:21:25.452317+0000","last_became_peered":"2024-03-04T16:21:25.452317+0000","last_unstale":"2024-03-04T16:24:09.012410+0000","last_undegraded":"2024-03-04T16:24:09.012410+0000","last_fullsized":"2024-03-04T16:24:09.012410+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:23:56.502927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:56.502927+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:24.900652+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":811,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009874+0000","last_change":"2024-03-04T16:23:09.663195+0000","last_active":"2024-03-04T16:24:09.009874+0000","last_peered":"2024-03-04T16:24:09.009874+0000","last_clean":"2024-03-04T16:24:09.009874+0000","last_became_active":"2024-03-04T16:20:07.486320+0000","last_became_peered":"2024-03-04T16:20:07.486320+0000","last_unstale":"2024-03-04T16:24:09.009874+0000","last_undegraded":"2024-03-04T16:24:09.009874+0000","last_fullsized":"2024-03-04T16:24:09.009874+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:09.662860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.662860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:13.759094+0000","scrub_duration":222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"338'11","reported_seq":1128,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.030741+0000","last_change":"2024-03-04T16:22:57.950248+0000","last_active":"2024-03-04T16:24:09.030741+0000","last_peered":"2024-03-04T16:24:09.030741+0000","last_clean":"2024-03-04T16:24:09.030741+0000","last_became_active":"2024-03-04T16:19:04.780989+0000","last_became_peered":"2024-03-04T16:19:04.780989+0000","last_unstale":"2024-03-04T16:24:09.030741+0000","last_undegraded":"2024-03-04T16:24:09.030741+0000","last_fullsized":"2024-03-04T16:24:09.030741+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:57.950104+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:22:57.950104+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"68'1","reported_seq":757,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996469+0000","last_change":"2024-03-04T16:22:00.982774+0000","last_active":"2024-03-04T16:24:08.996469+0000","last_peered":"2024-03-04T16:24:08.996469+0000","last_clean":"2024-03-04T16:24:08.996469+0000","last_became_active":"2024-03-04T16:07:36.244313+0000","last_became_peered":"2024-03-04T16:07:36.244313+0000","last_unstale":"2024-03-04T16:24:08.996469+0000","last_undegraded":"2024-03-04T16:24:08.996469+0000","last_fullsized":"2024-03-04T16:24:08.996469+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:00.982672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:00.982672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"441'21","reported_seq":892,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.011904+0000","last_change":"2024-03-04T16:22:56.048086+0000","last_active":"2024-03-04T16:24:09.011904+0000","last_peered":"2024-03-04T16:24:09.011904+0000","last_clean":"2024-03-04T16:24:09.011904+0000","last_became_active":"2024-03-04T16:02:45.744120+0000","last_became_peered":"2024-03-04T16:02:45.744120+0000","last_unstale":"2024-03-04T16:24:09.011904+0000","last_undegraded":"2024-03-04T16:24:09.011904+0000","last_fullsized":"2024-03-04T16:24:09.011904+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:22:56.047875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.047875+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.270041+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":275,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012208+0000","last_change":"2024-03-04T16:23:57.189430+0000","last_active":"2024-03-04T16:24:09.012208+0000","last_peered":"2024-03-04T16:24:09.012208+0000","last_clean":"2024-03-04T16:24:09.012208+0000","last_became_active":"2024-03-04T16:20:07.491512+0000","last_became_peered":"2024-03-04T16:20:07.491512+0000","last_unstale":"2024-03-04T16:24:09.012208+0000","last_undegraded":"2024-03-04T16:24:09.012208+0000","last_fullsized":"2024-03-04T16:24:09.012208+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:57.189312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:57.189312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:57.883657+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":839,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.023726+0000","last_change":"2024-03-04T16:22:41.839730+0000","last_active":"2024-03-04T16:24:09.023726+0000","last_peered":"2024-03-04T16:24:09.023726+0000","last_clean":"2024-03-04T16:24:09.023726+0000","last_became_active":"2024-03-04T16:02:47.774975+0000","last_became_peered":"2024-03-04T16:02:47.774975+0000","last_unstale":"2024-03-04T16:24:09.023726+0000","last_undegraded":"2024-03-04T16:24:09.023726+0000","last_fullsized":"2024-03-04T16:24:09.023726+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:41.839621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:41.839621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":882,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996371+0000","last_change":"2024-03-04T16:24:06.389501+0000","last_active":"2024-03-04T16:24:08.996371+0000","last_peered":"2024-03-04T16:24:08.996371+0000","last_clean":"2024-03-04T16:24:08.996371+0000","last_became_active":"2024-03-04T16:21:45.414717+0000","last_became_peered":"2024-03-04T16:21:45.414717+0000","last_unstale":"2024-03-04T16:24:08.996371+0000","last_undegraded":"2024-03-04T16:24:08.996371+0000","last_fullsized":"2024-03-04T16:24:08.996371+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:06.389376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:19:30.702539+0000","last_clean_scrub_stamp":"2024-03-04T16:24:06.389376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:27.498559+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'21","reported_seq":735,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.032191+0000","last_change":"2024-03-04T16:23:57.631940+0000","last_active":"2024-03-04T16:24:09.032191+0000","last_peered":"2024-03-04T16:24:09.032191+0000","last_clean":"2024-03-04T16:24:09.032191+0000","last_became_active":"2024-03-04T16:07:36.233445+0000","last_became_peered":"2024-03-04T16:07:36.233445+0000","last_unstale":"2024-03-04T16:24:09.032191+0000","last_undegraded":"2024-03-04T16:24:09.032191+0000","last_fullsized":"2024-03-04T16:24:09.032191+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:23:57.631810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:57.631810+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:07.370399+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996842+0000","last_change":"2024-03-04T16:21:45.412628+0000","last_active":"2024-03-04T16:24:08.996842+0000","last_peered":"2024-03-04T16:24:08.996842+0000","last_clean":"2024-03-04T16:24:08.996842+0000","last_became_active":"2024-03-04T16:21:45.411986+0000","last_became_peered":"2024-03-04T16:21:45.411986+0000","last_unstale":"2024-03-04T16:24:08.996842+0000","last_undegraded":"2024-03-04T16:24:08.996842+0000","last_fullsized":"2024-03-04T16:24:08.996842+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:20:59.610360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:20:59.610360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":854,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:11.943829+0000","last_change":"2024-03-04T16:24:11.943852+0000","last_active":"2024-03-04T16:24:11.943829+0000","last_peered":"2024-03-04T16:24:11.943829+0000","last_clean":"2024-03-04T16:24:11.943829+0000","last_became_active":"2024-03-04T16:02:47.769826+0000","last_became_peered":"2024-03-04T16:02:47.769826+0000","last_unstale":"2024-03-04T16:24:11.943829+0000","last_undegraded":"2024-03-04T16:24:11.943829+0000","last_fullsized":"2024-03-04T16:24:11.943829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:11.943763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:24:11.943763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:34.111273+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490786+0000","last_change":"2024-03-04T16:24:14.071010+0000","last_active":"2024-03-04T16:24:17.490786+0000","last_peered":"2024-03-04T16:24:17.490786+0000","last_clean":"2024-03-04T16:24:17.490786+0000","last_became_active":"2024-03-04T16:21:45.413989+0000","last_became_peered":"2024-03-04T16:21:45.413989+0000","last_unstale":"2024-03-04T16:24:17.490786+0000","last_undegraded":"2024-03-04T16:24:17.490786+0000","last_fullsized":"2024-03-04T16:24:17.490786+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:14.070762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:24:14.070762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:17.631770+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'21","reported_seq":873,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.025075+0000","last_change":"2024-03-04T16:22:44.928303+0000","last_active":"2024-03-04T16:24:09.025075+0000","last_peered":"2024-03-04T16:24:09.025075+0000","last_clean":"2024-03-04T16:24:09.025075+0000","last_became_active":"2024-03-04T16:02:45.750853+0000","last_became_peered":"2024-03-04T16:02:45.750853+0000","last_unstale":"2024-03-04T16:24:09.025075+0000","last_undegraded":"2024-03-04T16:24:09.025075+0000","last_fullsized":"2024-03-04T16:24:09.025075+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'15","last_scrub_stamp":"2024-03-04T16:22:44.928118+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.928118+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"458'11231","reported_seq":11367,"reported_epoch":458,"state":"active+clean+scrubbing","last_fresh":"2024-03-04T16:24:13.166078+0000","last_change":"2024-03-04T16:24:12.933495+0000","last_active":"2024-03-04T16:24:13.166078+0000","last_peered":"2024-03-04T16:24:13.166078+0000","last_clean":"2024-03-04T16:24:13.166078+0000","last_became_active":"2024-03-04T16:21:25.449029+0000","last_became_peered":"2024-03-04T16:21:25.449029+0000","last_unstale":"2024-03-04T16:24:13.166078+0000","last_undegraded":"2024-03-04T16:24:13.166078+0000","last_fullsized":"2024-03-04T16:24:13.166078+0000","mapping_epoch":401,"log_start":"443'6300","ondisk_log_start":"443'6300","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":699,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138084352,"num_objects":2107,"num_object_clones":0,"num_object_copies":6321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11231,"num_write_kb":426816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009380+0000","last_change":"2024-03-04T16:23:04.261493+0000","last_active":"2024-03-04T16:24:09.009380+0000","last_peered":"2024-03-04T16:24:09.009380+0000","last_clean":"2024-03-04T16:24:09.009380+0000","last_became_active":"2024-03-04T16:21:45.424279+0000","last_became_peered":"2024-03-04T16:21:45.424279+0000","last_unstale":"2024-03-04T16:24:09.009380+0000","last_undegraded":"2024-03-04T16:24:09.009380+0000","last_fullsized":"2024-03-04T16:24:09.009380+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.261358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.261358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.709711+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":912,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997308+0000","last_change":"2024-03-04T16:21:07.575776+0000","last_active":"2024-03-04T16:24:08.997308+0000","last_peered":"2024-03-04T16:24:08.997308+0000","last_clean":"2024-03-04T16:24:08.997308+0000","last_became_active":"2024-03-04T16:20:07.493115+0000","last_became_peered":"2024-03-04T16:20:07.493115+0000","last_unstale":"2024-03-04T16:24:08.997308+0000","last_undegraded":"2024-03-04T16:24:08.997308+0000","last_fullsized":"2024-03-04T16:24:08.997308+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:21:07.575695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:21:07.575695+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":986,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.008725+0000","last_change":"2024-03-04T16:23:00.064299+0000","last_active":"2024-03-04T16:24:09.008725+0000","last_peered":"2024-03-04T16:24:09.008725+0000","last_clean":"2024-03-04T16:24:09.008725+0000","last_became_active":"2024-03-04T16:21:45.422336+0000","last_became_peered":"2024-03-04T16:21:45.422336+0000","last_unstale":"2024-03-04T16:24:09.008725+0000","last_undegraded":"2024-03-04T16:24:09.008725+0000","last_fullsized":"2024-03-04T16:24:09.008725+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:23:00.064148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.064148+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.044607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"412'11","reported_seq":1160,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996594+0000","last_change":"2024-03-04T16:23:07.273194+0000","last_active":"2024-03-04T16:24:08.996594+0000","last_peered":"2024-03-04T16:24:08.996594+0000","last_clean":"2024-03-04T16:24:08.996594+0000","last_became_active":"2024-03-04T16:21:45.441048+0000","last_became_peered":"2024-03-04T16:21:45.441048+0000","last_unstale":"2024-03-04T16:24:08.996594+0000","last_undegraded":"2024-03-04T16:24:08.996594+0000","last_fullsized":"2024-03-04T16:24:08.996594+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'11","last_scrub_stamp":"2024-03-04T16:23:07.273057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.273057+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:35.024574+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"458'11111","reported_seq":11310,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.309554+0000","last_change":"2024-03-04T16:23:06.658833+0000","last_active":"2024-03-04T16:24:14.309554+0000","last_peered":"2024-03-04T16:24:14.309554+0000","last_clean":"2024-03-04T16:24:14.309554+0000","last_became_active":"2024-03-04T16:21:25.448720+0000","last_became_peered":"2024-03-04T16:21:25.448720+0000","last_unstale":"2024-03-04T16:24:14.309554+0000","last_undegraded":"2024-03-04T16:24:14.309554+0000","last_fullsized":"2024-03-04T16:24:14.309554+0000","mapping_epoch":401,"log_start":"444'6400","ondisk_log_start":"444'6400","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'6580","last_scrub_stamp":"2024-03-04T16:23:06.658440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:06.658440+0000","objects_scrubbed":6489,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:28.080402+0000","scrub_duration":3051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135987200,"num_objects":2075,"num_object_clones":0,"num_object_copies":6225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11111,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":844,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.026499+0000","last_change":"2024-03-04T16:24:07.428517+0000","last_active":"2024-03-04T16:24:09.026499+0000","last_peered":"2024-03-04T16:24:09.026499+0000","last_clean":"2024-03-04T16:24:09.026499+0000","last_became_active":"2024-03-04T16:02:49.794685+0000","last_became_peered":"2024-03-04T16:02:49.794685+0000","last_unstale":"2024-03-04T16:24:09.026499+0000","last_undegraded":"2024-03-04T16:24:09.026499+0000","last_fullsized":"2024-03-04T16:24:09.026499+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:07.428110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:24:07.428110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:32.757943+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":878,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997151+0000","last_change":"2024-03-04T16:22:50.302556+0000","last_active":"2024-03-04T16:24:08.997151+0000","last_peered":"2024-03-04T16:24:08.997151+0000","last_clean":"2024-03-04T16:24:08.997151+0000","last_became_active":"2024-03-04T16:20:07.497627+0000","last_became_peered":"2024-03-04T16:20:07.497627+0000","last_unstale":"2024-03-04T16:24:08.997151+0000","last_undegraded":"2024-03-04T16:24:08.997151+0000","last_fullsized":"2024-03-04T16:24:08.997151+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.302482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.302482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":679,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:13.739689+0000","last_change":"2024-03-04T16:24:13.739754+0000","last_active":"2024-03-04T16:24:13.739689+0000","last_peered":"2024-03-04T16:24:13.739689+0000","last_clean":"2024-03-04T16:24:13.739689+0000","last_became_active":"2024-03-04T16:21:45.443501+0000","last_became_peered":"2024-03-04T16:21:45.443501+0000","last_unstale":"2024-03-04T16:24:13.739689+0000","last_undegraded":"2024-03-04T16:24:13.739689+0000","last_fullsized":"2024-03-04T16:24:13.739689+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:24:13.739589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:24:13.739589+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:23.816606+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":269,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.018175+0000","last_change":"2024-03-04T16:23:54.086898+0000","last_active":"2024-03-04T16:24:09.018175+0000","last_peered":"2024-03-04T16:24:09.018175+0000","last_clean":"2024-03-04T16:24:09.018175+0000","last_became_active":"2024-03-04T16:19:04.768239+0000","last_became_peered":"2024-03-04T16:19:04.768239+0000","last_unstale":"2024-03-04T16:24:09.018175+0000","last_undegraded":"2024-03-04T16:24:09.018175+0000","last_fullsized":"2024-03-04T16:24:09.018175+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:54.086758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:54.086758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:09.488886+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"458'11340","reported_seq":13531,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:12.780113+0000","last_change":"2024-03-04T16:21:59.472159+0000","last_active":"2024-03-04T16:24:12.780113+0000","last_peered":"2024-03-04T16:24:12.780113+0000","last_clean":"2024-03-04T16:24:12.780113+0000","last_became_active":"2024-03-04T16:21:59.469208+0000","last_became_peered":"2024-03-04T16:21:59.469208+0000","last_unstale":"2024-03-04T16:24:12.780113+0000","last_undegraded":"2024-03-04T16:24:12.780113+0000","last_fullsized":"2024-03-04T16:24:12.780113+0000","mapping_epoch":421,"log_start":"445'6600","ondisk_log_start":"445'6600","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140640256,"num_objects":2146,"num_object_clones":0,"num_object_copies":6438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11340,"num_write_kb":431552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1027,"num_bytes_recovered":67305472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":870,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996131+0000","last_change":"2024-03-04T16:21:44.902269+0000","last_active":"2024-03-04T16:24:08.996131+0000","last_peered":"2024-03-04T16:24:08.996131+0000","last_clean":"2024-03-04T16:24:08.996131+0000","last_became_active":"2024-03-04T16:19:04.749223+0000","last_became_peered":"2024-03-04T16:19:04.749223+0000","last_unstale":"2024-03-04T16:24:08.996131+0000","last_undegraded":"2024-03-04T16:24:08.996131+0000","last_fullsized":"2024-03-04T16:24:08.996131+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:44.902188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:44.902188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":269,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010309+0000","last_change":"2024-03-04T16:23:55.133477+0000","last_active":"2024-03-04T16:24:09.010309+0000","last_peered":"2024-03-04T16:24:09.010309+0000","last_clean":"2024-03-04T16:24:09.010309+0000","last_became_active":"2024-03-04T16:19:04.780445+0000","last_became_peered":"2024-03-04T16:19:04.780445+0000","last_unstale":"2024-03-04T16:24:09.010309+0000","last_undegraded":"2024-03-04T16:24:09.010309+0000","last_fullsized":"2024-03-04T16:24:09.010309+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:55.133347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:02.504302+0000","last_clean_scrub_stamp":"2024-03-04T16:23:55.133347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:06.082602+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":876,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997211+0000","last_change":"2024-03-04T16:21:59.968201+0000","last_active":"2024-03-04T16:24:08.997211+0000","last_peered":"2024-03-04T16:24:08.997211+0000","last_clean":"2024-03-04T16:24:08.997211+0000","last_became_active":"2024-03-04T16:21:45.411738+0000","last_became_peered":"2024-03-04T16:21:45.411738+0000","last_unstale":"2024-03-04T16:24:08.997211+0000","last_undegraded":"2024-03-04T16:24:08.997211+0000","last_fullsized":"2024-03-04T16:24:08.997211+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:59.968123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:24.829107+0000","last_clean_scrub_stamp":"2024-03-04T16:21:59.968123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'52","reported_seq":190,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.491079+0000","last_change":"2024-03-04T16:22:51.690727+0000","last_active":"2024-03-04T16:24:17.491079+0000","last_peered":"2024-03-04T16:24:17.491079+0000","last_clean":"2024-03-04T16:24:17.491079+0000","last_became_active":"2024-03-04T16:22:24.826414+0000","last_became_peered":"2024-03-04T16:22:24.826414+0000","last_unstale":"2024-03-04T16:24:17.491079+0000","last_undegraded":"2024-03-04T16:24:17.491079+0000","last_fullsized":"2024-03-04T16:24:17.491079+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:22:51.690576+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.690576+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"458'11346","reported_seq":13524,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:12.794389+0000","last_change":"2024-03-04T16:21:52.427898+0000","last_active":"2024-03-04T16:24:12.794389+0000","last_peered":"2024-03-04T16:24:12.794389+0000","last_clean":"2024-03-04T16:24:12.794389+0000","last_became_active":"2024-03-04T16:21:52.426535+0000","last_became_peered":"2024-03-04T16:21:52.426535+0000","last_unstale":"2024-03-04T16:24:12.794389+0000","last_undegraded":"2024-03-04T16:24:12.794389+0000","last_fullsized":"2024-03-04T16:24:12.794389+0000","mapping_epoch":414,"log_start":"445'6600","ondisk_log_start":"445'6600","created":401,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4746,"log_dups_size":0,"ondisk_log_size":4746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142344192,"num_objects":2172,"num_object_clones":0,"num_object_copies":6516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11346,"num_write_kb":432576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1022,"num_bytes_recovered":66977792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":745,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996208+0000","last_change":"2024-03-04T16:22:06.237495+0000","last_active":"2024-03-04T16:24:08.996208+0000","last_peered":"2024-03-04T16:24:08.996208+0000","last_clean":"2024-03-04T16:24:08.996208+0000","last_became_active":"2024-03-04T16:19:04.779570+0000","last_became_peered":"2024-03-04T16:19:04.779570+0000","last_unstale":"2024-03-04T16:24:08.996208+0000","last_undegraded":"2024-03-04T16:24:08.996208+0000","last_fullsized":"2024-03-04T16:24:08.996208+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:06.237403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:12.401385+0000","last_clean_scrub_stamp":"2024-03-04T16:22:06.237403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":275,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010061+0000","last_change":"2024-03-04T16:22:54.512261+0000","last_active":"2024-03-04T16:24:09.010061+0000","last_peered":"2024-03-04T16:24:09.010061+0000","last_clean":"2024-03-04T16:24:09.010061+0000","last_became_active":"2024-03-04T16:20:07.487240+0000","last_became_peered":"2024-03-04T16:20:07.487240+0000","last_unstale":"2024-03-04T16:24:09.010061+0000","last_undegraded":"2024-03-04T16:24:09.010061+0000","last_fullsized":"2024-03-04T16:24:09.010061+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:54.512143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.512143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":875,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009625+0000","last_change":"2024-03-04T16:23:03.094410+0000","last_active":"2024-03-04T16:24:09.009625+0000","last_peered":"2024-03-04T16:24:09.009625+0000","last_clean":"2024-03-04T16:24:09.009625+0000","last_became_active":"2024-03-04T16:21:45.422860+0000","last_became_peered":"2024-03-04T16:21:45.422860+0000","last_unstale":"2024-03-04T16:24:09.009625+0000","last_undegraded":"2024-03-04T16:24:09.009625+0000","last_fullsized":"2024-03-04T16:24:09.009625+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.094190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.094190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.574130+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"441'25","reported_seq":768,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.031594+0000","last_change":"2024-03-04T16:22:56.668460+0000","last_active":"2024-03-04T16:24:09.031594+0000","last_peered":"2024-03-04T16:24:09.031594+0000","last_clean":"2024-03-04T16:24:09.031594+0000","last_became_active":"2024-03-04T16:21:10.564447+0000","last_became_peered":"2024-03-04T16:21:10.564447+0000","last_unstale":"2024-03-04T16:24:09.031594+0000","last_undegraded":"2024-03-04T16:24:09.031594+0000","last_fullsized":"2024-03-04T16:24:09.031594+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:22:56.668180+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.668180+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"458'11404","reported_seq":11587,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.480359+0000","last_change":"2024-03-04T16:22:42.339861+0000","last_active":"2024-03-04T16:24:14.480359+0000","last_peered":"2024-03-04T16:24:14.480359+0000","last_clean":"2024-03-04T16:24:14.480359+0000","last_became_active":"2024-03-04T16:21:25.449702+0000","last_became_peered":"2024-03-04T16:21:25.449702+0000","last_unstale":"2024-03-04T16:24:14.480359+0000","last_undegraded":"2024-03-04T16:24:14.480359+0000","last_fullsized":"2024-03-04T16:24:14.480359+0000","mapping_epoch":401,"log_start":"445'6900","ondisk_log_start":"445'6900","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5208","last_scrub_stamp":"2024-03-04T16:22:42.339680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.339680+0000","objects_scrubbed":5130,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2541,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131334144,"num_objects":2004,"num_object_clones":0,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11404,"num_write_kb":429056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":848,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.023224+0000","last_change":"2024-03-04T16:22:52.913981+0000","last_active":"2024-03-04T16:24:09.023224+0000","last_peered":"2024-03-04T16:24:09.023224+0000","last_clean":"2024-03-04T16:24:09.023224+0000","last_became_active":"2024-03-04T16:21:45.423440+0000","last_became_peered":"2024-03-04T16:21:45.423440+0000","last_unstale":"2024-03-04T16:24:09.023224+0000","last_undegraded":"2024-03-04T16:24:09.023224+0000","last_fullsized":"2024-03-04T16:24:09.023224+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:52.913842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:52.913842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":895,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009200+0000","last_change":"2024-03-04T16:23:54.067102+0000","last_active":"2024-03-04T16:24:09.009200+0000","last_peered":"2024-03-04T16:24:09.009200+0000","last_clean":"2024-03-04T16:24:09.009200+0000","last_became_active":"2024-03-04T16:21:45.425232+0000","last_became_peered":"2024-03-04T16:21:45.425232+0000","last_unstale":"2024-03-04T16:24:09.009200+0000","last_undegraded":"2024-03-04T16:24:09.009200+0000","last_fullsized":"2024-03-04T16:24:09.009200+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:23:54.066992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:23:54.066992+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:54.946510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490565+0000","last_change":"2024-03-04T16:22:45.604898+0000","last_active":"2024-03-04T16:24:17.490565+0000","last_peered":"2024-03-04T16:24:17.490565+0000","last_clean":"2024-03-04T16:24:17.490565+0000","last_became_active":"2024-03-04T16:21:45.437679+0000","last_became_peered":"2024-03-04T16:21:45.437679+0000","last_unstale":"2024-03-04T16:24:17.490565+0000","last_undegraded":"2024-03-04T16:24:17.490565+0000","last_fullsized":"2024-03-04T16:24:17.490565+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.604769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.604769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"441'55","reported_seq":1296,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.029890+0000","last_change":"2024-03-04T16:24:07.675742+0000","last_active":"2024-03-04T16:24:09.029890+0000","last_peered":"2024-03-04T16:24:09.029890+0000","last_clean":"2024-03-04T16:24:09.029890+0000","last_became_active":"2024-03-04T16:19:04.769086+0000","last_became_peered":"2024-03-04T16:19:04.769086+0000","last_unstale":"2024-03-04T16:24:09.029890+0000","last_undegraded":"2024-03-04T16:24:09.029890+0000","last_fullsized":"2024-03-04T16:24:09.029890+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'55","last_scrub_stamp":"2024-03-04T16:24:07.675601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:24:07.675601+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:24.412593+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":720,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.029997+0000","last_change":"2024-03-04T16:24:01.545874+0000","last_active":"2024-03-04T16:24:09.029997+0000","last_peered":"2024-03-04T16:24:09.029997+0000","last_clean":"2024-03-04T16:24:09.029997+0000","last_became_active":"2024-03-04T16:21:45.434502+0000","last_became_peered":"2024-03-04T16:21:45.434502+0000","last_unstale":"2024-03-04T16:24:09.029997+0000","last_undegraded":"2024-03-04T16:24:09.029997+0000","last_fullsized":"2024-03-04T16:24:09.029997+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:01.545767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:01.545767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:12.428526+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"459'11442","reported_seq":13723,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.637176+0000","last_change":"2024-03-04T16:24:11.654536+0000","last_active":"2024-03-04T16:24:17.637176+0000","last_peered":"2024-03-04T16:24:17.637176+0000","last_clean":"2024-03-04T16:24:17.637176+0000","last_became_active":"2024-03-04T16:22:04.786993+0000","last_became_peered":"2024-03-04T16:22:04.786993+0000","last_unstale":"2024-03-04T16:24:17.637176+0000","last_undegraded":"2024-03-04T16:24:17.637176+0000","last_fullsized":"2024-03-04T16:24:17.637176+0000","mapping_epoch":424,"log_start":"445'6600","ondisk_log_start":"445'6600","created":401,"last_epoch_clean":425,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'11048","last_scrub_stamp":"2024-03-04T16:24:11.654445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:24:11.654445+0000","objects_scrubbed":2251,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:31.740198+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120455168,"num_objects":1838,"num_object_clones":0,"num_object_copies":5514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11442,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1054,"num_bytes_recovered":69074944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":871,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997305+0000","last_change":"2024-03-04T16:21:45.415808+0000","last_active":"2024-03-04T16:24:08.997305+0000","last_peered":"2024-03-04T16:24:08.997305+0000","last_clean":"2024-03-04T16:24:08.997305+0000","last_became_active":"2024-03-04T16:21:45.415473+0000","last_became_peered":"2024-03-04T16:21:45.415473+0000","last_unstale":"2024-03-04T16:24:08.997305+0000","last_undegraded":"2024-03-04T16:24:08.997305+0000","last_fullsized":"2024-03-04T16:24:08.997305+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:40.973478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:21:40.973478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":889,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.006407+0000","last_change":"2024-03-04T16:22:51.222795+0000","last_active":"2024-03-04T16:24:09.006407+0000","last_peered":"2024-03-04T16:24:09.006407+0000","last_clean":"2024-03-04T16:24:09.006407+0000","last_became_active":"2024-03-04T16:21:45.437084+0000","last_became_peered":"2024-03-04T16:21:45.437084+0000","last_unstale":"2024-03-04T16:24:09.006407+0000","last_undegraded":"2024-03-04T16:24:09.006407+0000","last_fullsized":"2024-03-04T16:24:09.006407+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:51.222677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:51.222677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.054940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":269,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.017913+0000","last_change":"2024-03-04T16:22:09.686124+0000","last_active":"2024-03-04T16:24:09.017913+0000","last_peered":"2024-03-04T16:24:09.017913+0000","last_clean":"2024-03-04T16:24:09.017913+0000","last_became_active":"2024-03-04T16:19:04.780845+0000","last_became_peered":"2024-03-04T16:19:04.780845+0000","last_unstale":"2024-03-04T16:24:09.017913+0000","last_undegraded":"2024-03-04T16:24:09.017913+0000","last_fullsized":"2024-03-04T16:24:09.017913+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:09.685989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:22:09.685989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":514,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.021842+0000","last_change":"2024-03-04T16:23:08.368485+0000","last_active":"2024-03-04T16:24:09.021842+0000","last_peered":"2024-03-04T16:24:09.021842+0000","last_clean":"2024-03-04T16:24:09.021842+0000","last_became_active":"2024-03-04T16:12:35.720606+0000","last_became_peered":"2024-03-04T16:12:35.720606+0000","last_unstale":"2024-03-04T16:24:09.021842+0000","last_undegraded":"2024-03-04T16:24:09.021842+0000","last_fullsized":"2024-03-04T16:24:09.021842+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-04T16:23:08.368095+0000","last_deep_scrub":"90'18","last_deep_scrub_stamp":"2024-03-04T16:04:17.030053+0000","last_clean_scrub_stamp":"2024-03-04T16:23:08.368095+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.910733+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":269,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009044+0000","last_change":"2024-03-04T16:22:43.622761+0000","last_active":"2024-03-04T16:24:09.009044+0000","last_peered":"2024-03-04T16:24:09.009044+0000","last_clean":"2024-03-04T16:24:09.009044+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:24:09.009044+0000","last_undegraded":"2024-03-04T16:24:09.009044+0000","last_fullsized":"2024-03-04T16:24:09.009044+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:43.622383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:43.622383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"459'11400","reported_seq":13548,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.555155+0000","last_change":"2024-03-04T16:23:57.749401+0000","last_active":"2024-03-04T16:24:17.555155+0000","last_peered":"2024-03-04T16:24:17.555155+0000","last_clean":"2024-03-04T16:24:17.555155+0000","last_became_active":"2024-03-04T16:21:58.469058+0000","last_became_peered":"2024-03-04T16:21:58.469058+0000","last_unstale":"2024-03-04T16:24:17.555155+0000","last_undegraded":"2024-03-04T16:24:17.555155+0000","last_fullsized":"2024-03-04T16:24:17.555155+0000","mapping_epoch":420,"log_start":"445'6500","ondisk_log_start":"445'6500","created":401,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'9991","last_scrub_stamp":"2024-03-04T16:23:57.749265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:57.749265+0000","objects_scrubbed":3310,"log_size":4900,"log_dups_size":0,"ondisk_log_size":4900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:57.918634+0000","scrub_duration":712,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123207680,"num_objects":1880,"num_object_clones":0,"num_object_copies":5640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11400,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":984,"num_bytes_recovered":64487424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":857,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010241+0000","last_change":"2024-03-04T16:22:50.355727+0000","last_active":"2024-03-04T16:24:09.010241+0000","last_peered":"2024-03-04T16:24:09.010241+0000","last_clean":"2024-03-04T16:24:09.010241+0000","last_became_active":"2024-03-04T16:20:07.500640+0000","last_became_peered":"2024-03-04T16:20:07.500640+0000","last_unstale":"2024-03-04T16:24:09.010241+0000","last_undegraded":"2024-03-04T16:24:09.010241+0000","last_fullsized":"2024-03-04T16:24:09.010241+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:50.355598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:50.355598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:18.959365+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1187,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996232+0000","last_change":"2024-03-04T16:22:48.307804+0000","last_active":"2024-03-04T16:24:08.996232+0000","last_peered":"2024-03-04T16:24:08.996232+0000","last_clean":"2024-03-04T16:24:08.996232+0000","last_became_active":"2024-03-04T16:21:48.434375+0000","last_became_peered":"2024-03-04T16:21:48.434375+0000","last_unstale":"2024-03-04T16:24:08.996232+0000","last_undegraded":"2024-03-04T16:24:08.996232+0000","last_fullsized":"2024-03-04T16:24:08.996232+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:22:48.307675+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.307675+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":795,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009156+0000","last_change":"2024-03-04T16:23:55.041669+0000","last_active":"2024-03-04T16:24:09.009156+0000","last_peered":"2024-03-04T16:24:09.009156+0000","last_clean":"2024-03-04T16:24:09.009156+0000","last_became_active":"2024-03-04T16:21:45.424708+0000","last_became_peered":"2024-03-04T16:21:45.424708+0000","last_unstale":"2024-03-04T16:24:09.009156+0000","last_undegraded":"2024-03-04T16:24:09.009156+0000","last_fullsized":"2024-03-04T16:24:09.009156+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:55.041559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:23:55.041559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:06.847347+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"441'51","reported_seq":186,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.492827+0000","last_change":"2024-03-04T16:23:51.050483+0000","last_active":"2024-03-04T16:24:17.492827+0000","last_peered":"2024-03-04T16:24:17.492827+0000","last_clean":"2024-03-04T16:24:17.492827+0000","last_became_active":"2024-03-04T16:21:45.411357+0000","last_became_peered":"2024-03-04T16:21:45.411357+0000","last_unstale":"2024-03-04T16:24:17.492827+0000","last_undegraded":"2024-03-04T16:24:17.492827+0000","last_fullsized":"2024-03-04T16:24:17.492827+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'51","last_scrub_stamp":"2024-03-04T16:23:51.050351+0000","last_deep_scrub":"391'38","last_deep_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_clean_scrub_stamp":"2024-03-04T16:23:51.050351+0000","objects_scrubbed":8,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:06.142506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":877,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997718+0000","last_change":"2024-03-04T16:22:04.052615+0000","last_active":"2024-03-04T16:24:08.997718+0000","last_peered":"2024-03-04T16:24:08.997718+0000","last_clean":"2024-03-04T16:24:08.997718+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:24:08.997718+0000","last_undegraded":"2024-03-04T16:24:08.997718+0000","last_fullsized":"2024-03-04T16:24:08.997718+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:04.052531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:04.052531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"458'11281","reported_seq":13565,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.469361+0000","last_change":"2024-03-04T16:23:56.613541+0000","last_active":"2024-03-04T16:24:14.469361+0000","last_peered":"2024-03-04T16:24:14.469361+0000","last_clean":"2024-03-04T16:24:14.469361+0000","last_became_active":"2024-03-04T16:21:55.408063+0000","last_became_peered":"2024-03-04T16:21:55.408063+0000","last_unstale":"2024-03-04T16:24:14.469361+0000","last_undegraded":"2024-03-04T16:24:14.469361+0000","last_fullsized":"2024-03-04T16:24:14.469361+0000","mapping_epoch":417,"log_start":"445'6800","ondisk_log_start":"445'6800","created":401,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'10020","last_scrub_stamp":"2024-03-04T16:23:56.613269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:56.613269+0000","objects_scrubbed":3281,"log_size":4481,"log_dups_size":0,"ondisk_log_size":4481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:04.435889+0000","scrub_duration":1520,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129171456,"num_objects":1971,"num_object_clones":0,"num_object_copies":5913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11281,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":69009408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":861,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012508+0000","last_change":"2024-03-04T16:23:03.390202+0000","last_active":"2024-03-04T16:24:09.012508+0000","last_peered":"2024-03-04T16:24:09.012508+0000","last_clean":"2024-03-04T16:24:09.012508+0000","last_became_active":"2024-03-04T16:02:49.791399+0000","last_became_peered":"2024-03-04T16:02:49.791399+0000","last_unstale":"2024-03-04T16:24:09.012508+0000","last_undegraded":"2024-03-04T16:24:09.012508+0000","last_fullsized":"2024-03-04T16:24:09.012508+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:03.389804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:52.944410+0000","last_clean_scrub_stamp":"2024-03-04T16:23:03.389804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.921181+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":722,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.035669+0000","last_change":"2024-03-04T16:24:02.526868+0000","last_active":"2024-03-04T16:24:09.035669+0000","last_peered":"2024-03-04T16:24:09.035669+0000","last_clean":"2024-03-04T16:24:09.035669+0000","last_became_active":"2024-03-04T16:20:07.498576+0000","last_became_peered":"2024-03-04T16:20:07.498576+0000","last_unstale":"2024-03-04T16:24:09.035669+0000","last_undegraded":"2024-03-04T16:24:09.035669+0000","last_fullsized":"2024-03-04T16:24:09.035669+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:02.526663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:24:02.526663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:04.438841+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":2167,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:10.926712+0000","last_change":"2024-03-04T16:21:41.881246+0000","last_active":"2024-03-04T16:24:10.926712+0000","last_peered":"2024-03-04T16:24:10.926712+0000","last_clean":"2024-03-04T16:24:10.926712+0000","last_became_active":"2024-03-04T16:02:42.677345+0000","last_became_peered":"2024-03-04T16:02:42.677345+0000","last_unstale":"2024-03-04T16:24:10.926712+0000","last_undegraded":"2024-03-04T16:24:10.926712+0000","last_fullsized":"2024-03-04T16:24:10.926712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:21:41.881125+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:21:41.881125+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1281,"num_read_kb":1276,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":818,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:10.651376+0000","last_change":"2024-03-04T16:24:10.651432+0000","last_active":"2024-03-04T16:24:10.651376+0000","last_peered":"2024-03-04T16:24:10.651376+0000","last_clean":"2024-03-04T16:24:10.651376+0000","last_became_active":"2024-03-04T16:22:24.838293+0000","last_became_peered":"2024-03-04T16:22:24.838293+0000","last_unstale":"2024-03-04T16:24:10.651376+0000","last_undegraded":"2024-03-04T16:24:10.651376+0000","last_fullsized":"2024-03-04T16:24:10.651376+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-04T16:24:10.651300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:24:10.651300+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:35.965654+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":863,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010512+0000","last_change":"2024-03-04T16:22:45.128630+0000","last_active":"2024-03-04T16:24:09.010512+0000","last_peered":"2024-03-04T16:24:09.010512+0000","last_clean":"2024-03-04T16:24:09.010512+0000","last_became_active":"2024-03-04T16:02:47.768592+0000","last_became_peered":"2024-03-04T16:02:47.768592+0000","last_unstale":"2024-03-04T16:24:09.010512+0000","last_undegraded":"2024-03-04T16:24:09.010512+0000","last_fullsized":"2024-03-04T16:24:09.010512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:45.128548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:22:45.128548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"458'11262","reported_seq":11472,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.461281+0000","last_change":"2024-03-04T16:23:53.898834+0000","last_active":"2024-03-04T16:24:14.461281+0000","last_peered":"2024-03-04T16:24:14.461281+0000","last_clean":"2024-03-04T16:24:14.461281+0000","last_became_active":"2024-03-04T16:21:25.448277+0000","last_became_peered":"2024-03-04T16:21:25.448277+0000","last_unstale":"2024-03-04T16:24:14.461281+0000","last_undegraded":"2024-03-04T16:24:14.461281+0000","last_fullsized":"2024-03-04T16:24:14.461281+0000","mapping_epoch":401,"log_start":"445'7000","ondisk_log_start":"445'7000","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"454'9876","last_scrub_stamp":"2024-03-04T16:23:53.898498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:53.898498+0000","objects_scrubbed":3447,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:23.002547+0000","scrub_duration":1411,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132382720,"num_objects":2020,"num_object_clones":0,"num_object_copies":6060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11262,"num_write_kb":425024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":885,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:12.847757+0000","last_change":"2024-03-04T16:24:12.847797+0000","last_active":"2024-03-04T16:24:12.847757+0000","last_peered":"2024-03-04T16:24:12.847757+0000","last_clean":"2024-03-04T16:24:12.847757+0000","last_became_active":"2024-03-04T16:21:45.442233+0000","last_became_peered":"2024-03-04T16:21:45.442233+0000","last_unstale":"2024-03-04T16:24:12.847757+0000","last_undegraded":"2024-03-04T16:24:12.847757+0000","last_fullsized":"2024-03-04T16:24:12.847757+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:12.847651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:09.376470+0000","last_clean_scrub_stamp":"2024-03-04T16:24:12.847651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:13.157593+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":869,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012736+0000","last_change":"2024-03-04T16:22:48.112648+0000","last_active":"2024-03-04T16:24:09.012736+0000","last_peered":"2024-03-04T16:24:09.012736+0000","last_clean":"2024-03-04T16:24:09.012736+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:24:09.012736+0000","last_undegraded":"2024-03-04T16:24:09.012736+0000","last_fullsized":"2024-03-04T16:24:09.012736+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.112512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.112512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'47","reported_seq":854,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009303+0000","last_change":"2024-03-04T16:22:49.132475+0000","last_active":"2024-03-04T16:24:09.009303+0000","last_peered":"2024-03-04T16:24:09.009303+0000","last_clean":"2024-03-04T16:24:09.009303+0000","last_became_active":"2024-03-04T16:07:36.221153+0000","last_became_peered":"2024-03-04T16:07:36.221153+0000","last_unstale":"2024-03-04T16:24:09.009303+0000","last_undegraded":"2024-03-04T16:24:09.009303+0000","last_fullsized":"2024-03-04T16:24:09.009303+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'37","last_scrub_stamp":"2024-03-04T16:22:49.132304+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:22:49.132304+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490952+0000","last_change":"2024-03-04T16:22:08.771819+0000","last_active":"2024-03-04T16:24:17.490952+0000","last_peered":"2024-03-04T16:24:17.490952+0000","last_clean":"2024-03-04T16:24:17.490952+0000","last_became_active":"2024-03-04T16:21:45.412152+0000","last_became_peered":"2024-03-04T16:21:45.412152+0000","last_unstale":"2024-03-04T16:24:17.490952+0000","last_undegraded":"2024-03-04T16:24:17.490952+0000","last_fullsized":"2024-03-04T16:24:17.490952+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_clean_scrub_stamp":"2024-03-04T16:22:08.770293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"458'11186","reported_seq":13449,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:13.096928+0000","last_change":"2024-03-04T16:22:54.725063+0000","last_active":"2024-03-04T16:24:13.096928+0000","last_peered":"2024-03-04T16:24:13.096928+0000","last_clean":"2024-03-04T16:24:13.096928+0000","last_became_active":"2024-03-04T16:21:54.483830+0000","last_became_peered":"2024-03-04T16:21:54.483830+0000","last_unstale":"2024-03-04T16:24:13.096928+0000","last_undegraded":"2024-03-04T16:24:13.096928+0000","last_fullsized":"2024-03-04T16:24:13.096928+0000","mapping_epoch":416,"log_start":"439'5600","ondisk_log_start":"439'5600","created":401,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"442'5897","last_scrub_stamp":"2024-03-04T16:22:54.724570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:54.724570+0000","objects_scrubbed":5842,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560088,"num_objects":2100,"num_object_clones":0,"num_object_copies":6300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2100,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11186,"num_write_kb":425089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1031,"num_bytes_recovered":67567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":873,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.028045+0000","last_change":"2024-03-04T16:22:47.720482+0000","last_active":"2024-03-04T16:24:09.028045+0000","last_peered":"2024-03-04T16:24:09.028045+0000","last_clean":"2024-03-04T16:24:09.028045+0000","last_became_active":"2024-03-04T16:19:04.834912+0000","last_became_peered":"2024-03-04T16:19:04.834912+0000","last_unstale":"2024-03-04T16:24:09.028045+0000","last_undegraded":"2024-03-04T16:24:09.028045+0000","last_fullsized":"2024-03-04T16:24:09.028045+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:47.720278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:47.720278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":882,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010074+0000","last_change":"2024-03-04T16:24:06.959214+0000","last_active":"2024-03-04T16:24:09.010074+0000","last_peered":"2024-03-04T16:24:09.010074+0000","last_clean":"2024-03-04T16:24:09.010074+0000","last_became_active":"2024-03-04T16:19:04.762915+0000","last_became_peered":"2024-03-04T16:19:04.762915+0000","last_unstale":"2024-03-04T16:24:09.010074+0000","last_undegraded":"2024-03-04T16:24:09.010074+0000","last_fullsized":"2024-03-04T16:24:09.010074+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:06.959089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_clean_scrub_stamp":"2024-03-04T16:24:06.959089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:07.788973+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":937,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.020234+0000","last_change":"2024-03-04T16:24:06.632149+0000","last_active":"2024-03-04T16:24:09.020234+0000","last_peered":"2024-03-04T16:24:09.020234+0000","last_clean":"2024-03-04T16:24:09.020234+0000","last_became_active":"2024-03-04T16:20:07.504270+0000","last_became_peered":"2024-03-04T16:20:07.504270+0000","last_unstale":"2024-03-04T16:24:09.020234+0000","last_undegraded":"2024-03-04T16:24:09.020234+0000","last_fullsized":"2024-03-04T16:24:09.020234+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:24:06.632021+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:24:06.632021+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:35.751616+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"412'12","reported_seq":154,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490655+0000","last_change":"2024-03-04T16:23:09.792730+0000","last_active":"2024-03-04T16:24:17.490655+0000","last_peered":"2024-03-04T16:24:17.490655+0000","last_clean":"2024-03-04T16:24:17.490655+0000","last_became_active":"2024-03-04T16:21:45.414139+0000","last_became_peered":"2024-03-04T16:21:45.414139+0000","last_unstale":"2024-03-04T16:24:17.490655+0000","last_undegraded":"2024-03-04T16:24:17.490655+0000","last_fullsized":"2024-03-04T16:24:17.490655+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'12","last_scrub_stamp":"2024-03-04T16:23:09.792648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:09.792648+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"458'11364","reported_seq":11562,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.432321+0000","last_change":"2024-03-04T16:23:12.469661+0000","last_active":"2024-03-04T16:24:14.432321+0000","last_peered":"2024-03-04T16:24:14.432321+0000","last_clean":"2024-03-04T16:24:14.432321+0000","last_became_active":"2024-03-04T16:21:25.451006+0000","last_became_peered":"2024-03-04T16:21:25.451006+0000","last_unstale":"2024-03-04T16:24:14.432321+0000","last_undegraded":"2024-03-04T16:24:14.432321+0000","last_fullsized":"2024-03-04T16:24:14.432321+0000","mapping_epoch":401,"log_start":"445'7100","ondisk_log_start":"445'7100","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7132","last_scrub_stamp":"2024-03-04T16:23:12.469418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.469418+0000","objects_scrubbed":6443,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:15.303121+0000","scrub_duration":3140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138018816,"num_objects":2106,"num_object_clones":0,"num_object_copies":6318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11364,"num_write_kb":431040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":885,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.996966+0000","last_change":"2024-03-04T16:23:10.418223+0000","last_active":"2024-03-04T16:24:08.996966+0000","last_peered":"2024-03-04T16:24:08.996966+0000","last_clean":"2024-03-04T16:24:08.996966+0000","last_became_active":"2024-03-04T16:21:45.413068+0000","last_became_peered":"2024-03-04T16:21:45.413068+0000","last_unstale":"2024-03-04T16:24:08.996966+0000","last_undegraded":"2024-03-04T16:24:08.996966+0000","last_fullsized":"2024-03-04T16:24:08.996966+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:10.418117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.418117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:27.395028+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"68'1","reported_seq":722,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.031990+0000","last_change":"2024-03-04T16:22:53.683732+0000","last_active":"2024-03-04T16:24:09.031990+0000","last_peered":"2024-03-04T16:24:09.031990+0000","last_clean":"2024-03-04T16:24:09.031990+0000","last_became_active":"2024-03-04T16:20:07.494182+0000","last_became_peered":"2024-03-04T16:20:07.494182+0000","last_unstale":"2024-03-04T16:24:09.031990+0000","last_undegraded":"2024-03-04T16:24:09.031990+0000","last_fullsized":"2024-03-04T16:24:09.031990+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:22:53.683136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:53.683136+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'49","reported_seq":722,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.008843+0000","last_change":"2024-03-04T16:23:07.122889+0000","last_active":"2024-03-04T16:24:09.008843+0000","last_peered":"2024-03-04T16:24:09.008843+0000","last_clean":"2024-03-04T16:24:09.008843+0000","last_became_active":"2024-03-04T16:21:45.423250+0000","last_became_peered":"2024-03-04T16:21:45.423250+0000","last_unstale":"2024-03-04T16:24:09.008843+0000","last_undegraded":"2024-03-04T16:24:09.008843+0000","last_fullsized":"2024-03-04T16:24:09.008843+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'49","last_scrub_stamp":"2024-03-04T16:23:07.122781+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.122781+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.026776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"366'9","reported_seq":1126,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.030289+0000","last_change":"2024-03-04T16:24:08.121724+0000","last_active":"2024-03-04T16:24:09.030289+0000","last_peered":"2024-03-04T16:24:09.030289+0000","last_clean":"2024-03-04T16:24:09.030289+0000","last_became_active":"2024-03-04T16:20:07.484919+0000","last_became_peered":"2024-03-04T16:20:07.484919+0000","last_unstale":"2024-03-04T16:24:09.030289+0000","last_undegraded":"2024-03-04T16:24:09.030289+0000","last_fullsized":"2024-03-04T16:24:09.030289+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:24:08.121611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:08.121611+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:10.758658+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"458'11426","reported_seq":11550,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:12.826613+0000","last_change":"2024-03-04T16:21:25.449756+0000","last_active":"2024-03-04T16:24:12.826613+0000","last_peered":"2024-03-04T16:24:12.826613+0000","last_clean":"2024-03-04T16:24:12.826613+0000","last_became_active":"2024-03-04T16:21:25.449469+0000","last_became_peered":"2024-03-04T16:21:25.449469+0000","last_unstale":"2024-03-04T16:24:12.826613+0000","last_undegraded":"2024-03-04T16:24:12.826613+0000","last_fullsized":"2024-03-04T16:24:12.826613+0000","mapping_epoch":401,"log_start":"445'6700","ondisk_log_start":"445'6700","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:21:24.414280+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139984896,"num_objects":2136,"num_object_clones":0,"num_object_copies":6408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11426,"num_write_kb":433984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":816,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.024085+0000","last_change":"2024-03-04T16:23:04.878400+0000","last_active":"2024-03-04T16:24:09.024085+0000","last_peered":"2024-03-04T16:24:09.024085+0000","last_clean":"2024-03-04T16:24:09.024085+0000","last_became_active":"2024-03-04T16:21:45.424326+0000","last_became_peered":"2024-03-04T16:21:45.424326+0000","last_unstale":"2024-03-04T16:24:09.024085+0000","last_undegraded":"2024-03-04T16:24:09.024085+0000","last_fullsized":"2024-03-04T16:24:09.024085+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:04.878278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:04.878278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:30.211352+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":319,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:08.997157+0000","last_change":"2024-03-04T16:22:42.336778+0000","last_active":"2024-03-04T16:24:08.997157+0000","last_peered":"2024-03-04T16:24:08.997157+0000","last_clean":"2024-03-04T16:24:08.997157+0000","last_became_active":"2024-03-04T16:21:25.438797+0000","last_became_peered":"2024-03-04T16:21:25.438797+0000","last_unstale":"2024-03-04T16:24:08.997157+0000","last_undegraded":"2024-03-04T16:24:08.997157+0000","last_fullsized":"2024-03-04T16:24:08.997157+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:42.336656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.336656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"441'42","reported_seq":969,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.024127+0000","last_change":"2024-03-04T16:23:07.898360+0000","last_active":"2024-03-04T16:24:09.024127+0000","last_peered":"2024-03-04T16:24:09.024127+0000","last_clean":"2024-03-04T16:24:09.024127+0000","last_became_active":"2024-03-04T16:07:36.220863+0000","last_became_peered":"2024-03-04T16:07:36.220863+0000","last_unstale":"2024-03-04T16:24:09.024127+0000","last_undegraded":"2024-03-04T16:24:09.024127+0000","last_fullsized":"2024-03-04T16:24:09.024127+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-04T16:23:07.898070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:07.898070+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:32.268466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":215,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:11.182123+0000","last_change":"2024-03-04T16:24:11.182156+0000","last_active":"2024-03-04T16:24:11.182123+0000","last_peered":"2024-03-04T16:24:11.182123+0000","last_clean":"2024-03-04T16:24:11.182123+0000","last_became_active":"2024-03-04T16:20:07.487202+0000","last_became_peered":"2024-03-04T16:20:07.487202+0000","last_unstale":"2024-03-04T16:24:11.182123+0000","last_undegraded":"2024-03-04T16:24:11.182123+0000","last_fullsized":"2024-03-04T16:24:11.182123+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:11.182046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:11.182046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:24.161377+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"458'11205","reported_seq":11389,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:14.438840+0000","last_change":"2024-03-04T16:22:42.067136+0000","last_active":"2024-03-04T16:24:14.438840+0000","last_peered":"2024-03-04T16:24:14.438840+0000","last_clean":"2024-03-04T16:24:14.438840+0000","last_became_active":"2024-03-04T16:21:25.450165+0000","last_became_peered":"2024-03-04T16:21:25.450165+0000","last_unstale":"2024-03-04T16:24:14.438840+0000","last_undegraded":"2024-03-04T16:24:14.438840+0000","last_fullsized":"2024-03-04T16:24:14.438840+0000","mapping_epoch":401,"log_start":"445'6900","ondisk_log_start":"445'6900","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'5014","last_scrub_stamp":"2024-03-04T16:22:42.066893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:42.066893+0000","objects_scrubbed":4968,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134152192,"num_objects":2047,"num_object_clones":0,"num_object_copies":6141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11205,"num_write_kb":424064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":855,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009809+0000","last_change":"2024-03-04T16:22:46.243647+0000","last_active":"2024-03-04T16:24:09.009809+0000","last_peered":"2024-03-04T16:24:09.009809+0000","last_clean":"2024-03-04T16:24:09.009809+0000","last_became_active":"2024-03-04T16:07:36.221654+0000","last_became_peered":"2024-03-04T16:07:36.221654+0000","last_unstale":"2024-03-04T16:24:09.009809+0000","last_undegraded":"2024-03-04T16:24:09.009809+0000","last_fullsized":"2024-03-04T16:24:09.009809+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:46.243413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:46.243413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"366'9","reported_seq":285,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010268+0000","last_change":"2024-03-04T16:23:02.057648+0000","last_active":"2024-03-04T16:24:09.010268+0000","last_peered":"2024-03-04T16:24:09.010268+0000","last_clean":"2024-03-04T16:24:09.010268+0000","last_became_active":"2024-03-04T16:20:07.484334+0000","last_became_peered":"2024-03-04T16:20:07.484334+0000","last_unstale":"2024-03-04T16:24:09.010268+0000","last_undegraded":"2024-03-04T16:24:09.010268+0000","last_fullsized":"2024-03-04T16:24:09.010268+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'9","last_scrub_stamp":"2024-03-04T16:23:02.057434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:02.057434+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:14.009367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":856,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.024033+0000","last_change":"2024-03-04T16:22:48.920668+0000","last_active":"2024-03-04T16:24:09.024033+0000","last_peered":"2024-03-04T16:24:09.024033+0000","last_clean":"2024-03-04T16:24:09.024033+0000","last_became_active":"2024-03-04T16:02:43.719363+0000","last_became_peered":"2024-03-04T16:02:43.719363+0000","last_unstale":"2024-03-04T16:24:09.024033+0000","last_undegraded":"2024-03-04T16:24:09.024033+0000","last_fullsized":"2024-03-04T16:24:09.024033+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:48.920550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:22:48.920550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'46","reported_seq":950,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.022030+0000","last_change":"2024-03-04T16:23:00.358198+0000","last_active":"2024-03-04T16:24:09.022030+0000","last_peered":"2024-03-04T16:24:09.022030+0000","last_clean":"2024-03-04T16:24:09.022030+0000","last_became_active":"2024-03-04T16:20:07.507258+0000","last_became_peered":"2024-03-04T16:20:07.507258+0000","last_unstale":"2024-03-04T16:24:09.022030+0000","last_undegraded":"2024-03-04T16:24:09.022030+0000","last_fullsized":"2024-03-04T16:24:09.022030+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:23:00.358056+0000","last_deep_scrub":"301'34","last_deep_scrub_stamp":"2024-03-04T16:17:05.452417+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.358056+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:24.797848+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"459'11687","reported_seq":13666,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.623501+0000","last_change":"2024-03-04T16:24:02.926540+0000","last_active":"2024-03-04T16:24:17.623501+0000","last_peered":"2024-03-04T16:24:17.623501+0000","last_clean":"2024-03-04T16:24:17.623501+0000","last_became_active":"2024-03-04T16:21:56.407370+0000","last_became_peered":"2024-03-04T16:21:56.407370+0000","last_unstale":"2024-03-04T16:24:17.623501+0000","last_undegraded":"2024-03-04T16:24:17.623501+0000","last_fullsized":"2024-03-04T16:24:17.623501+0000","mapping_epoch":418,"log_start":"445'6800","ondisk_log_start":"445'6800","created":401,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'10642","last_scrub_stamp":"2024-03-04T16:24:02.925574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:24:02.925574+0000","objects_scrubbed":2917,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:21.820158+0000","scrub_duration":789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121176064,"num_objects":1849,"num_object_clones":0,"num_object_copies":5547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11687,"num_write_kb":433152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1068,"num_bytes_recovered":69992448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"75'1","reported_seq":124,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.490997+0000","last_change":"2024-03-04T16:23:00.740759+0000","last_active":"2024-03-04T16:24:17.490997+0000","last_peered":"2024-03-04T16:24:17.490997+0000","last_clean":"2024-03-04T16:24:17.490997+0000","last_became_active":"2024-03-04T16:21:45.420505+0000","last_became_peered":"2024-03-04T16:21:45.420505+0000","last_unstale":"2024-03-04T16:24:17.490997+0000","last_undegraded":"2024-03-04T16:24:17.490997+0000","last_fullsized":"2024-03-04T16:24:17.490997+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'1","last_scrub_stamp":"2024-03-04T16:23:00.740631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:23:00.740631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:25.137776+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":879,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010501+0000","last_change":"2024-03-04T16:23:12.207020+0000","last_active":"2024-03-04T16:24:09.010501+0000","last_peered":"2024-03-04T16:24:09.010501+0000","last_clean":"2024-03-04T16:24:09.010501+0000","last_became_active":"2024-03-04T16:21:45.425391+0000","last_became_peered":"2024-03-04T16:21:45.425391+0000","last_unstale":"2024-03-04T16:24:09.010501+0000","last_undegraded":"2024-03-04T16:24:09.010501+0000","last_fullsized":"2024-03-04T16:24:09.010501+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.206943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.206943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:38.403715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":824,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.018102+0000","last_change":"2024-03-04T16:24:08.091492+0000","last_active":"2024-03-04T16:24:09.018102+0000","last_peered":"2024-03-04T16:24:09.018102+0000","last_clean":"2024-03-04T16:24:09.018102+0000","last_became_active":"2024-03-04T16:20:07.471591+0000","last_became_peered":"2024-03-04T16:20:07.471591+0000","last_unstale":"2024-03-04T16:24:09.018102+0000","last_undegraded":"2024-03-04T16:24:09.018102+0000","last_fullsized":"2024-03-04T16:24:09.018102+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:08.091383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:29.654032+0000","last_clean_scrub_stamp":"2024-03-04T16:24:08.091383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:22.775298+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"441'32","reported_seq":144,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.491000+0000","last_change":"2024-03-04T16:23:05.801666+0000","last_active":"2024-03-04T16:24:17.491000+0000","last_peered":"2024-03-04T16:24:17.491000+0000","last_clean":"2024-03-04T16:24:17.491000+0000","last_became_active":"2024-03-04T16:21:45.446219+0000","last_became_peered":"2024-03-04T16:21:45.446219+0000","last_unstale":"2024-03-04T16:24:17.491000+0000","last_undegraded":"2024-03-04T16:24:17.491000+0000","last_fullsized":"2024-03-04T16:24:17.491000+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:23:05.801531+0000","last_deep_scrub":"113'16","last_deep_scrub_stamp":"2024-03-04T16:05:41.187699+0000","last_clean_scrub_stamp":"2024-03-04T16:23:05.801531+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"458'11377","reported_seq":11575,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:13.083768+0000","last_change":"2024-03-04T16:22:58.382258+0000","last_active":"2024-03-04T16:24:13.083768+0000","last_peered":"2024-03-04T16:24:13.083768+0000","last_clean":"2024-03-04T16:24:13.083768+0000","last_became_active":"2024-03-04T16:21:25.445028+0000","last_became_peered":"2024-03-04T16:21:25.445028+0000","last_unstale":"2024-03-04T16:24:13.083768+0000","last_undegraded":"2024-03-04T16:24:13.083768+0000","last_fullsized":"2024-03-04T16:24:13.083768+0000","mapping_epoch":401,"log_start":"441'5800","ondisk_log_start":"441'5800","created":401,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'6281","last_scrub_stamp":"2024-03-04T16:22:58.381858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.381858+0000","objects_scrubbed":6184,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146604032,"num_objects":2237,"num_object_clones":0,"num_object_copies":6711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11377,"num_write_kb":435648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":289,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:10.410082+0000","last_change":"2024-03-04T16:24:10.410124+0000","last_active":"2024-03-04T16:24:10.410082+0000","last_peered":"2024-03-04T16:24:10.410082+0000","last_clean":"2024-03-04T16:24:10.410082+0000","last_became_active":"2024-03-04T16:21:45.423031+0000","last_became_peered":"2024-03-04T16:21:45.423031+0000","last_unstale":"2024-03-04T16:24:10.410082+0000","last_undegraded":"2024-03-04T16:24:10.410082+0000","last_fullsized":"2024-03-04T16:24:10.410082+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:10.409960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:10.409960+0000","last_clean_scrub_stamp":"2024-03-04T16:24:10.409960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:30.608034+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":876,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.012936+0000","last_change":"2024-03-04T16:24:04.583295+0000","last_active":"2024-03-04T16:24:09.012936+0000","last_peered":"2024-03-04T16:24:09.012936+0000","last_clean":"2024-03-04T16:24:09.012936+0000","last_became_active":"2024-03-04T16:21:45.434815+0000","last_became_peered":"2024-03-04T16:21:45.434815+0000","last_unstale":"2024-03-04T16:24:09.012936+0000","last_undegraded":"2024-03-04T16:24:09.012936+0000","last_fullsized":"2024-03-04T16:24:09.012936+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:24:04.583170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:24:04.583170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:10.140362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":120,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.491143+0000","last_change":"2024-03-04T16:23:01.769342+0000","last_active":"2024-03-04T16:24:17.491143+0000","last_peered":"2024-03-04T16:24:17.491143+0000","last_clean":"2024-03-04T16:24:17.491143+0000","last_became_active":"2024-03-04T16:21:45.423438+0000","last_became_peered":"2024-03-04T16:21:45.423438+0000","last_unstale":"2024-03-04T16:24:17.491143+0000","last_undegraded":"2024-03-04T16:24:17.491143+0000","last_fullsized":"2024-03-04T16:24:17.491143+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:01.769222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:23:01.769222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":974,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009472+0000","last_change":"2024-03-04T16:23:10.470966+0000","last_active":"2024-03-04T16:24:09.009472+0000","last_peered":"2024-03-04T16:24:09.009472+0000","last_clean":"2024-03-04T16:24:09.009472+0000","last_became_active":"2024-03-04T16:21:45.439301+0000","last_became_peered":"2024-03-04T16:21:45.439301+0000","last_unstale":"2024-03-04T16:24:09.009472+0000","last_undegraded":"2024-03-04T16:24:09.009472+0000","last_fullsized":"2024-03-04T16:24:09.009472+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-04T16:23:10.470699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:23:10.470699+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.738180+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"459'11682","reported_seq":13468,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-04T16:24:17.633013+0000","last_change":"2024-03-04T16:24:08.820866+0000","last_active":"2024-03-04T16:24:17.633013+0000","last_peered":"2024-03-04T16:24:17.633013+0000","last_clean":"2024-03-04T16:24:17.633013+0000","last_became_active":"2024-03-04T16:21:59.471792+0000","last_became_peered":"2024-03-04T16:21:59.471792+0000","last_unstale":"2024-03-04T16:24:17.633013+0000","last_undegraded":"2024-03-04T16:24:17.633013+0000","last_fullsized":"2024-03-04T16:24:17.633013+0000","mapping_epoch":421,"log_start":"445'6800","ondisk_log_start":"445'6800","created":401,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"457'11093","last_scrub_stamp":"2024-03-04T16:24:08.820586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:24.414280+0000","last_clean_scrub_stamp":"2024-03-04T16:24:08.820586+0000","objects_scrubbed":2460,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:25:37.323419+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120717312,"num_objects":1842,"num_object_clones":0,"num_object_copies":5526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11682,"num_write_kb":432768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1070,"num_bytes_recovered":70123520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":804,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.013109+0000","last_change":"2024-03-04T16:22:44.238092+0000","last_active":"2024-03-04T16:24:09.013109+0000","last_peered":"2024-03-04T16:24:09.013109+0000","last_clean":"2024-03-04T16:24:09.013109+0000","last_became_active":"2024-03-04T16:20:07.480213+0000","last_became_peered":"2024-03-04T16:20:07.480213+0000","last_unstale":"2024-03-04T16:24:09.013109+0000","last_undegraded":"2024-03-04T16:24:09.013109+0000","last_fullsized":"2024-03-04T16:24:09.013109+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:22:44.237641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:22:44.237641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":335,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.009621+0000","last_change":"2024-03-04T16:22:58.485899+0000","last_active":"2024-03-04T16:24:09.009621+0000","last_peered":"2024-03-04T16:24:09.009621+0000","last_clean":"2024-03-04T16:24:09.009621+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:24:09.009621+0000","last_undegraded":"2024-03-04T16:24:09.009621+0000","last_fullsized":"2024-03-04T16:24:09.009621+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:22:58.485608+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:22:58.485608+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":142,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.010078+0000","last_change":"2024-03-04T16:23:12.437298+0000","last_active":"2024-03-04T16:24:09.010078+0000","last_peered":"2024-03-04T16:24:09.010078+0000","last_clean":"2024-03-04T16:24:09.010078+0000","last_became_active":"2024-03-04T16:21:24.444615+0000","last_became_peered":"2024-03-04T16:21:24.444615+0000","last_unstale":"2024-03-04T16:24:09.010078+0000","last_undegraded":"2024-03-04T16:24:09.010078+0000","last_fullsized":"2024-03-04T16:24:09.010078+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:23:12.437183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:59.774151+0000","last_clean_scrub_stamp":"2024-03-04T16:23:12.437183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:24:17.188381+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":856,"reported_epoch":458,"state":"active+clean","last_fresh":"2024-03-04T16:24:09.022975+0000","last_change":"2024-03-04T16:22:56.926525+0000","last_active":"2024-03-04T16:24:09.022975+0000","last_peered":"2024-03-04T16:24:09.022975+0000","last_clean":"2024-03-04T16:24:09.022975+0000","last_became_active":"2024-03-04T16:21:45.425802+0000","last_became_peered":"2024-03-04T16:21:45.425802+0000","last_unstale":"2024-03-04T16:24:09.022975+0000","last_undegraded":"2024-03-04T16:24:09.022975+0000","last_fullsized":"2024-03-04T16:24:09.022975+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:22:56.926373+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:22:56.926373+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2131820568,"num_objects":32530,"num_object_clones":0,"num_object_copies":97590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32530,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181744,"num_write_kb":6856705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8309,"num_bytes_recovered":544538624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6446198784,"data_stored":6446186568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76344,"ondisk_log_size":76344,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1281,"num_read_kb":1276,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4246,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":8906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1051,"ondisk_log_size":1051,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070726,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1928452,"kb_used_data":986556,"kb_used_omap":1,"kb_used_meta":941886,"kb_avail":91816700,"statfs":{"total":95995035648,"available":94020300800,"internally_reserved":0,"allocated":1010233344,"data_stored":1009848054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":964491722},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365067,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1377668,"kb_used_data":959164,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92367484,"statfs":{"total":95995035648,"available":94584303616,"internally_reserved":0,"allocated":982183936,"data_stored":981797196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":428538310},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575140,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317152,"kb_used_data":1207256,"kb_used_omap":1,"kb_used_meta":1109886,"kb_avail":91428000,"statfs":{"total":95995035648,"available":93622272000,"internally_reserved":0,"allocated":1236230144,"data_stored":1235814838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1136523717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738664,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1848320,"kb_used_data":950776,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91896832,"statfs":{"total":95995035648,"available":94102355968,"internally_reserved":0,"allocated":973594624,"data_stored":973212772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460076,"kb_used_data":824996,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92285076,"statfs":{"total":95995035648,"available":94499917824,"internally_reserved":0,"allocated":844795904,"data_stored":844433032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2129104,"kb_used_data":931336,"kb_used_omap":1,"kb_used_meta":1197694,"kb_avail":91616048,"statfs":{"total":95995035648,"available":93814833152,"internally_reserved":0,"allocated":953688064,"data_stored":953289460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1226439112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626704,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579760,"kb_used_data":551784,"kb_used_omap":1,"kb_used_meta":1027966,"kb_avail":92165392,"statfs":{"total":95995035648,"available":94377361408,"internally_reserved":0,"allocated":565026816,"data_stored":564659650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1052637638},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":946274304,"data_stored":946274304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1204293632,"data_stored":1204289560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":843452416,"data_stored":843448344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":939720704,"data_stored":939720704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":559677440,"data_stored":559677440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":959844352,"data_stored":959840280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":992935936,"data_stored":992935936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:24:22.355 INFO:tasks.thrashosds.thrasher:Setting 6.6 to [5, 7, 2, 1, 6, 0] 2024-03-04T16:24:22.356 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.6', '5', '7', '2', '1', '6', '0'] 2024-03-04T16:24:22.356 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pg-upmap-items 6.6 5 7 2 1 6 0 2024-03-04T16:24:22.438 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:22.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:22.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:22.596 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:22.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:22.698 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:22.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:22.864 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:22.893 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:22.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:22.928 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-04T16:24:22.970 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:22 smithi038 ceph-mon[29779]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:22.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:22 smithi038 ceph-mon[38113]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:22.995 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:23.010 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:22 smithi176 ceph-mon[33619]: pgmap v1012: 148 pgs: 148 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:23.024 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:23.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:24:23.126 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:23.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:23.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.303 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:23.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:23.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:23.426 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:23.452 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:23.554 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.599 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:23.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:23.662 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:23.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:23.746 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:23.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:24:23.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:24:23.803 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:24:23.804 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:24:23.822 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:23.832 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:24:23.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:23.876 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:24:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:23.900 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:23.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:23.924 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1532965298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:23.925 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:23.925 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:23.925 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1532965298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:23.925 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:23.925 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:23.925 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:24:23.960 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:24:23.978 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:24:24.001 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:24.027 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:23 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1532965298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:23 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:24.128 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:24.155 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:24.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.306 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:24.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:24.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.447 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:24.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:24.548 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.584 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:24.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:24.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:24.725 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:24.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:24.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:24 smithi176 ceph-mon[33619]: pgmap v1013: 148 pgs: 148 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:24.868 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:24.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:24.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:24.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:24.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:24 smithi038 ceph-mon[29779]: pgmap v1013: 148 pgs: 148 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:24.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:24 smithi038 ceph-mon[38113]: pgmap v1013: 148 pgs: 148 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:25.001 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:25.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:25.103 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.173 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:25.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:25.174 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:25.232 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:25.274 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.325 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:25.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:25.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.471 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:25.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:25.572 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.611 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:25.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178682 10.16 10:688de60d:::benchmark_data_smithi038_155391_object42725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 310.23542846100003, 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96282650700000005, 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.654 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178682 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414379+0000", 2024-03-04T16:24:25.655 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414380+0000", 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2559999999999999e-06 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414385+0000", 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0080000000000002e-06 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.656 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414395+0000", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.961e-06 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414402+0000", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3019999999999997e-06 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414455+0000", 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2580999999999997e-05 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.657 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.414460+0000", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4310000000000002e-06 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371209+0000", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95674911100000004 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371328+0000", 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118875 2024-03-04T16:24:25.658 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.371653+0000", 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00032488900000000003 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377063+0000", 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054099769999999998 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.659 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377128+0000", 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4801000000000005e-05 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377138+0000", 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.006e-05 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377178+0000", 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9690000000000001e-05 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.660 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377205+0000", 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7565000000000001e-05 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178701 10.16 10:6aa3b8c6:::benchmark_data_smithi038_155391_object42744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 310.21601814500002, 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94355677400000004, 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.661 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178701 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.662 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433789+0000", 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433791+0000", 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.294e-06 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433798+0000", 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7100000000000001e-06 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.663 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433813+0000", 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5096000000000001e-05 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433819+0000", 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2659999999999998e-06 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.664 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433886+0000", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7197000000000001e-05 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.433895+0000", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3100000000000001e-06 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.373812+0000", 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93991698599999995 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.665 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.373926+0000", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011376599999999999 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.374218+0000", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00029243200000000002 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377099+0000", 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0028806830000000002 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.666 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377255+0000", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015678600000000001 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377284+0000", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8600999999999999e-05 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377320+0000", 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5969000000000001e-05 2024-03-04T16:24:25.667 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.377346+0000", 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5678e-05 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:178708 10.16 10:6808437f:::benchmark_data_smithi038_155391_object42751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 310.21294726799999, 2024-03-04T16:24:25.668 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94346813900000004, 2024-03-04T16:24:25.669 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.669 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.669 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.669 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 178708 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.670 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436860+0000", 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436862+0000", 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0260000000000002e-06 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436868+0000", 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2219999999999999e-06 2024-03-04T16:24:25.671 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436884+0000", 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5888000000000001e-05 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436892+0000", 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.3750000000000003e-06 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.672 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436957+0000", 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.4307999999999995e-05 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:15.436968+0000", 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1086e-05 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.673 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375734+0000", 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93876637100000004 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375815+0000", 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0813e-05 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,4", 2024-03-04T16:24:25.674 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.375991+0000", 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017593200000000001 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.378898+0000", 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0029075699999999999 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380128+0000", 2024-03-04T16:24:25.675 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00122941 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380235+0000", 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000107013 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.676 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380295+0000", 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0449999999999999e-05 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:16.380328+0000", 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2675000000000001e-05 2024-03-04T16:24:25.677 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183512 10.9 10:94a4b12e:::benchmark_data_smithi038_155391_object47555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 304.26730578000002, 2024-03-04T16:24:25.678 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.022642394, 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183512 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.679 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382501+0000", 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382503+0000", 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.962e-06 2024-03-04T16:24:25.680 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382508+0000", 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2329999999999997e-06 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.681 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382521+0000", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2875e-05 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382529+0000", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8199999999999997e-06 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382589+0000", 2024-03-04T16:24:25.682 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9803999999999999e-05 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.382698+0000", 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000108675 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,4,7", 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.383077+0000", 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00037943100000000001 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.683 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.684 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386465+0000", 2024-03-04T16:24:25.684 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033882270000000002 2024-03-04T16:24:25.684 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.684 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.684 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.688 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386614+0000", 2024-03-04T16:24:25.688 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014898 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.386666+0000", 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2080999999999998e-05 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.390710+0000", 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040436980000000001 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.689 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.390725+0000", 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4959e-05 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405084+0000", 2024-03-04T16:24:25.690 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0143585209999999 2024-03-04T16:24:25.691 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.692 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.692 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.692 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405144+0000", 2024-03-04T16:24:25.692 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0127999999999998e-05 2024-03-04T16:24:25.692 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183547 10.9 10:96435141:::benchmark_data_smithi038_155391_object47590:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:24:25.693 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 304.21774763100001, 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97993219899999995, 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.694 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183547 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.695 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.696 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:24:25.696 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.696 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.696 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.696 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432059+0000", 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.697 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432061+0000", 2024-03-04T16:24:25.698 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.052e-06 2024-03-04T16:24:25.698 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432068+0000", 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.2330000000000003e-06 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.699 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432108+0000", 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.9860000000000001e-05 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.700 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432116+0000", 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2120000000000003e-06 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432173+0000", 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7655000000000003e-05 2024-03-04T16:24:25.701 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.432181+0000", 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0029999999999994e-06 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405472+0000", 2024-03-04T16:24:25.702 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97329052000000005 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405592+0000", 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120238 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.703 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.405948+0000", 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035554 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.704 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411767+0000", 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0058191359999999999 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411869+0000", 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010232499999999999 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.705 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411887+0000", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7493999999999999e-05 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411941+0000", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4728999999999997e-05 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.411992+0000", 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0201999999999998e-05 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.706 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183589 10.9 10:90edb329:::benchmark_data_smithi038_155391_object47632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 304.17370033399999, 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93606389700000003, 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183589 2024-03-04T16:24:25.707 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.708 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476107+0000", 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476109+0000", 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9230000000000001e-06 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.709 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476114+0000", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3159999999999996e-06 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476124+0000", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0521e-05 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476132+0000", 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9300000000000003e-06 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.710 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476187+0000", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4250999999999999e-05 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.476196+0000", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0250000000000008e-06 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407549+0000", 2024-03-04T16:24:25.711 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93135321299999996 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407654+0000", 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105425 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.407911+0000", 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000256269 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.712 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412002+0000", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040916559999999999 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412051+0000", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8906000000000002e-05 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412095+0000", 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3649999999999997e-05 2024-03-04T16:24:25.713 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412132+0000", 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7221999999999997e-05 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412171+0000", 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8590000000000002e-05 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.714 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183607 10.9 10:9627eeb7:::benchmark_data_smithi038_155391_object47650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 304.15847099899997, 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93008667, 2024-03-04T16:24:25.715 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.716 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.716 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.716 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183607 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:24:25.717 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491336+0000", 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491338+0000", 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7990000000000001e-06 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.718 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491343+0000", 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9649999999999999e-06 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491352+0000", 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4809999999999995e-06 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.719 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491360+0000", 2024-03-04T16:24:25.720 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1240000000000005e-06 2024-03-04T16:24:25.720 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.720 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.720 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.720 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491417+0000", 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6093e-05 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.491425+0000", 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2409999999999995e-06 2024-03-04T16:24:25.721 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.722 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.722 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.722 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.409749+0000", 2024-03-04T16:24:25.722 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.91832424599999996 2024-03-04T16:24:25.722 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.409898+0000", 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014876099999999999 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.723 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.724 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:24:25.724 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.410135+0000", 2024-03-04T16:24:25.724 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00023713799999999999 2024-03-04T16:24:25.724 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.724 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.725 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.725 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.414069+0000", 2024-03-04T16:24:25.725 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0039342309999999998 2024-03-04T16:24:25.725 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.725 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415821+0000", 2024-03-04T16:24:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0017515129999999999 2024-03-04T16:24:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.726 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.727 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.727 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421320+0000", 2024-03-04T16:24:25.727 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054993680000000001 2024-03-04T16:24:25.727 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.727 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.728 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.728 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421377+0000", 2024-03-04T16:24:25.728 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6845999999999997e-05 2024-03-04T16:24:25.728 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.728 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.729 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.729 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421423+0000", 2024-03-04T16:24:25.729 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5864e-05 2024-03-04T16:24:25.729 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.729 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:183622 10.9 10:93d2c9ee:::benchmark_data_smithi038_155391_object47665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:24:25.730 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 304.14449997499997, 2024-03-04T16:24:25.731 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.916269899, 2024-03-04T16:24:25.731 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.731 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.731 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.731 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.732 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.732 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 183622 2024-03-04T16:24:25.732 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.732 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.732 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.733 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.733 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:24:25.733 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.733 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.733 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.734 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.734 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505307+0000", 2024-03-04T16:24:25.734 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.734 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.734 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505309+0000", 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.483e-06 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.735 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.736 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505313+0000", 2024-03-04T16:24:25.736 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8400000000000002e-06 2024-03-04T16:24:25.736 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.736 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.737 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.737 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505325+0000", 2024-03-04T16:24:25.737 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1219e-05 2024-03-04T16:24:25.737 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.737 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.738 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.738 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505332+0000", 2024-03-04T16:24:25.738 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.515e-06 2024-03-04T16:24:25.738 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.738 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.739 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.739 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505392+0000", 2024-03-04T16:24:25.739 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0238000000000002e-05 2024-03-04T16:24:25.739 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.739 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.740 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.740 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:21.505408+0000", 2024-03-04T16:24:25.740 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5650000000000001e-05 2024-03-04T16:24:25.740 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.740 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.741 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.741 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412222+0000", 2024-03-04T16:24:25.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.90681353600000003 2024-03-04T16:24:25.741 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412356+0000", 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013452399999999999 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,7", 2024-03-04T16:24:25.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.412626+0000", 2024-03-04T16:24:25.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00026958 2024-03-04T16:24:25.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415835+0000", 2024-03-04T16:24:25.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0032093949999999999 2024-03-04T16:24:25.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.415886+0000", 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0495999999999999e-05 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421486+0000", 2024-03-04T16:24:25.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0056001439999999996 2024-03-04T16:24:25.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421542+0000", 2024-03-04T16:24:25.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5809e-05 2024-03-04T16:24:25.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:19:22.421577+0000", 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5469999999999998e-05 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223791 10.6 10:6309caeb:::benchmark_data_smithi038_155391_object87834:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:24:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:24:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 259.14158622899998, 2024-03-04T16:24:25.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5336528830000002, 2024-03-04T16:24:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223791 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:24:25.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508221+0000", 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508222+0000", 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-06 2024-03-04T16:24:25.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508227+0000", 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5580000000000002e-06 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508235+0000", 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.79e-06 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508240+0000", 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2850000000000002e-06 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508284+0000", 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.4307000000000001e-05 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.508301+0000", 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.6427e-05 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.461692+0000", 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95339159900000003 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.461707+0000", 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4426999999999999e-05 2024-03-04T16:24:25.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.795533+0000", 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.333826843 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.795668+0000", 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013503899999999999 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.756 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.796150+0000", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00048139299999999999 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.803054+0000", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.006903986 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.803411+0000", 2024-03-04T16:24:25.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035698300000000001 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.871742+0000", 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.068331431999999998 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041692+0000", 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.169949765 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041759+0000", 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7399999999999998e-05 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.041874+0000", 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000114273 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254087 10.16 10:6a7c6658:::benchmark_data_smithi038_155391_object118130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 217.182500108, 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99874777000000003, 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254087 2024-03-04T16:24:25.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467307+0000", 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467309+0000", 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.75e-06 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467314+0000", 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1569999999999999e-06 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467325+0000", 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1e-05 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467331+0000", 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7219999999999996e-06 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467390+0000", 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9556999999999999e-05 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.467401+0000", 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0740000000000001e-05 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457681+0000", 2024-03-04T16:24:25.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99027973999999996 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457779+0000", 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.8640000000000004e-05 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.457885+0000", 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000105567 2024-03-04T16:24:25.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465170+0000", 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0072846789999999996 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465304+0000", 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013415700000000001 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465960+0000", 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00065629800000000004 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.466018+0000", 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8063000000000003e-05 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.466055+0000", 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6699999999999998e-05 2024-03-04T16:24:25.767 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254092 10.16 10:69ffd62a:::benchmark_data_smithi038_155391_object118135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 217.17665678200001, 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99644071400000001, 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.768 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254092 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473150+0000", 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473152+0000", 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7549999999999999e-06 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473157+0000", 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.203e-06 2024-03-04T16:24:25.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473167+0000", 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9590000000000007e-06 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473174+0000", 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0489999999999998e-06 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473221+0000", 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6257000000000001e-05 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.473228+0000", 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5939999999999996e-06 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460640+0000", 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98741175400000003 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460760+0000", 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000120544 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.460882+0000", 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000121316 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465259+0000", 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0043770479999999997 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.468940+0000", 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003680853 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469484+0000", 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00054460999999999995 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469539+0000", 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4691999999999997e-05 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469591+0000", 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2080000000000003e-05 2024-03-04T16:24:25.775 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:254116 10.16 10:6e4c31d2:::benchmark_data_smithi038_155391_object118159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 217.15297605999999, 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97311181499999999, 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.776 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 254116 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496831+0000", 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496832+0000", 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3500000000000005e-07 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496836+0000", 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7290000000000002e-06 2024-03-04T16:24:25.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496844+0000", 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.5709999999999998e-06 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.496849+0000", 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9690000000000002e-06 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.498442+0000", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001592996 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:48.498448+0000", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4909999999999996e-06 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463072+0000", 2024-03-04T16:24:25.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96462448499999998 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463190+0000", 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118097 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.781 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.463330+0000", 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014000300000000001 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.465267+0000", 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019365389999999999 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.468974+0000", 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037075680000000001 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469853+0000", 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000878531 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469902+0000", 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9224999999999997e-05 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:49.469943+0000", 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0676000000000002e-05 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.785 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259082 10.6 10:66146a1d:::benchmark_data_smithi038_155391_object123125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 211.06647363799999, 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17653590499999999, 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259082 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583333+0000", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583335+0000", 2024-03-04T16:24:25.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5039999999999999e-06 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583339+0000", 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3059999999999999e-06 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583348+0000", 2024-03-04T16:24:25.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0659999999999993e-06 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583354+0000", 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.378e-06 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583400+0000", 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6449999999999997e-05 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583492+0000", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1613999999999994e-05 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583574+0000", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2582000000000003e-05 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587766+0000", 2024-03-04T16:24:25.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041915850000000003 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587819+0000", 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3044999999999998e-05 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.588257+0000", 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043791799999999998 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759743+0000", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.171485729 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759788+0000", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5698000000000002e-05 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.759869+0000", 2024-03-04T16:24:25.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1030000000000002e-05 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259084 10.6 10:649bf51c:::benchmark_data_smithi038_155391_object123127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 211.06633637300001, 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.180419145, 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.793 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259084 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583471+0000", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9400000000000005e-07 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583474+0000", 2024-03-04T16:24:25.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3549999999999999e-06 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583478+0000", 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6260000000000003e-06 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.583482+0000", 2024-03-04T16:24:25.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6679999999999999e-06 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585467+0000", 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019849939999999999 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585595+0000", 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012844000000000001 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.797 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.585735+0000", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013928899999999999 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.588323+0000", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025887800000000002 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.589825+0000", 2024-03-04T16:24:25.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015020369999999999 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763632+0000", 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17380702000000001 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763717+0000", 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4268e-05 2024-03-04T16:24:25.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763777+0000", 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0584000000000001e-05 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763890+0000", 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011249 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.800 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259086 10.11 10:8dc5ffb0:::benchmark_data_smithi038_155391_object123129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 211.065481318, 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.179469561, 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.801 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259086 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584326+0000", 2024-03-04T16:24:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584327+0000", 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9500000000000001e-07 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584330+0000", 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.117e-06 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584335+0000", 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6930000000000004e-06 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584341+0000", 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8479999999999997e-06 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.804 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584375+0000", 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3337999999999998e-05 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584457+0000", 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2729e-05 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,7", 2024-03-04T16:24:25.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.584576+0000", 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118279 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.586907+0000", 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023310290000000001 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587025+0000", 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118304 2024-03-04T16:24:25.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763696+0000", 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17667132799999999 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763753+0000", 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6672000000000002e-05 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763795+0000", 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2429000000000003e-05 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259089 10.16 10:6e6ec3de:::benchmark_data_smithi038_155391_object123132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 211.06279967099999, 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17683321099999999, 2024-03-04T16:24:25.808 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259089 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587007+0000", 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587009+0000", 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9920000000000002e-06 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587015+0000", 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3190000000000002e-06 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587026+0000", 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1127999999999999e-05 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587033+0000", 2024-03-04T16:24:25.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5499999999999997e-06 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587799+0000", 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00076608299999999998 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587902+0000", 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000102632 2024-03-04T16:24:25.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.587965+0000", 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.3398999999999994e-05 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.589944+0000", 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019787400000000001 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.590363+0000", 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000418928 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763652+0000", 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.17328922399999999 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763746+0000", 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3706e-05 2024-03-04T16:24:25.814 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:54.763841+0000", 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4510000000000001e-05 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:259608 10.16 10:6d95926b:::benchmark_data_smithi038_155391_object123651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:24:25.815 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 210.238786788, 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15055500899999999, 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 259608 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.816 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411020+0000", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411022+0000", 2024-03-04T16:24:25.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.637e-06 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411027+0000", 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4940000000000003e-06 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411041+0000", 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3390999999999999e-05 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411049+0000", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.2859999999999999e-06 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411108+0000", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8727999999999997e-05 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411197+0000", 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.9566000000000003e-05 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,2", 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.411272+0000", 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4139000000000003e-05 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.413150+0000", 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018781780000000001 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.413612+0000", 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000462058 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561402+0000", 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.147790011 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561525+0000", 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012313499999999999 2024-03-04T16:24:25.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:55.561575+0000", 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0386000000000002e-05 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:110159 11.7 11:ebfd3f21:::benchmark_data_smithi038_182448_object3020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:23:09.517985+0000", 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.131822330999995, 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.145085192, 2024-03-04T16:24:25.822 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 110159 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.517985+0000", 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.517985+0000", 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.517986+0000", 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5549999999999999e-06 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.517991+0000", 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9219999999999996e-06 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.518003+0000", 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1362000000000001e-05 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.518010+0000", 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.267e-06 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.518063+0000", 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3372999999999999e-05 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.518206+0000", 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000142309 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.518339+0000", 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00013377800000000001 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521393+0000", 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003053741 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.533721+0000", 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.01232812 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.662997+0000", 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.129275799 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.663035+0000", 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8148999999999997e-05 2024-03-04T16:24:25.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.663070+0000", 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.4817000000000002e-05 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:110165 11.7 11:ec9eef61:::benchmark_data_smithi038_182448_object3026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:23:09.521612+0000", 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 76.128195536000007, 2024-03-04T16:24:25.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.14156957100000001, 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 110165 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521612+0000", 2024-03-04T16:24:25.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521612+0000", 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521614+0000", 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.007e-06 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521623+0000", 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.4320000000000005e-06 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521632+0000", 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1530000000000003e-06 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521640+0000", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5190000000000003e-06 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521690+0000", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0424999999999999e-05 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521804+0000", 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0001136 2024-03-04T16:24:25.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.521874+0000", 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.0401999999999994e-05 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.524074+0000", 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0021996440000000002 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.538368+0000", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.014293807 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.663108+0000", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.124740566 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.663143+0000", 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.5280000000000001e-05 2024-03-04T16:24:25.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:23:09.663181+0000", 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7735999999999999e-05 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:166946 11.7 11:e6bce1e8:::benchmark_data_smithi038_182448_object59807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-04T16:24:25.836 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:24:01.372283+0000", 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 24.277524213, 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15993616299999999, 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 166946 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.837 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372283+0000", 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372283+0000", 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:25.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372290+0000", 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.861e-06 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372295+0000", 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5899999999999998e-06 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372308+0000", 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2218999999999999e-05 2024-03-04T16:24:25.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372315+0000", 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8089999999999993e-06 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372385+0000", 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9874000000000002e-05 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372504+0000", 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000118905 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.372607+0000", 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010272099999999999 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:25.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.375113+0000", 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0025060780000000001 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.375320+0000", 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00020670399999999999 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.532149+0000", 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15682943299999999 2024-03-04T16:24:25.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.532187+0000", 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.7883000000000003e-05 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:01.532219+0000", 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.2085999999999997e-05 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.843 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:25.844 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:25.844 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:24:25.871 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:25.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:25.973 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:26.004 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:26.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:26.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:26.252 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:26.283 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:26.385 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:26.517 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:26.543 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:26.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.675 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:26.776 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.801 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:26.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:26.902 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:26.945 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:26.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:27.046 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:27.076 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:27.161 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:24:27.177 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.261 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:27.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:27.362 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:27.392 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:27.409 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:24:27.493 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:27.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.524 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:27.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:27.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.676 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:27.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:27.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:27 smithi038 ceph-mon[29779]: pgmap v1014: 148 pgs: 148 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:27.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:27 smithi038 ceph-mon[38113]: pgmap v1014: 148 pgs: 148 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:27.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:27.781 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:27.816 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:27.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:27 smithi176 ceph-mon[33619]: pgmap v1014: 148 pgs: 148 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:27.918 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:27.946 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:27.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:28.048 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:28.083 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:28.184 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:28.218 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:28.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:28.352 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:28.453 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:28.482 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:28.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:28.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:28.624 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:28.726 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:28.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:28 smithi038 ceph-mon[29779]: pgmap v1015: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 995 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:24:28.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:28 smithi038 ceph-mon[38113]: pgmap v1015: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 995 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:24:28.754 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:28.856 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:28.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:28 smithi176 ceph-mon[33619]: pgmap v1015: 148 pgs: 148 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 995 B/s rd, 0 B/s wr, 1.21k op/s 2024-03-04T16:24:28.896 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:28.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:28.998 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:29.037 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:29.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:29.200 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:29.301 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:29.331 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:29.433 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:29.464 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:29.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.592 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:29.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:29.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:29.716 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:29.818 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:29.846 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:29.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:29.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:29.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:30.093 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:30.115 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:24:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:24:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:198341 11.3 11:c7f20daa:::benchmark_data_smithi038_182448_object91202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e459)", 2024-03-04T16:24:30.115 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:24:29.790350+0000", 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 0.003117149, 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003150333, 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "waiting for sub ops", 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 198341 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.116 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790350+0000", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790350+0000", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790352+0000", 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0229999999999999e-06 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.117 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790357+0000", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8409999999999999e-06 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790377+0000", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9970999999999999e-05 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790384+0000", 2024-03-04T16:24:30.118 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3849999999999996e-06 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790436+0000", 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1520999999999997e-05 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790559+0000", 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012312900000000001 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 1,7", 2024-03-04T16:24:30.119 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:24:29.790672+0000", 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112899 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: ], 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout: "num_ops": 1 2024-03-04T16:24:30.120 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:24:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.121 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:30.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:30.222 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:30.248 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:30.350 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:30.395 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:30.497 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.526 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:30.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:30.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:30.668 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:30.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:30.804 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:24:30.809 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:30.859 INFO:teuthology.orchestra.run.smithi038.stderr:set 6.6 pg_upmap_items mapping to [5->7,2->1,6->0] 2024-03-04T16:24:30.911 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:30.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:30.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:31.055 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.100 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:31.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:30 smithi176 ceph-mon[33619]: pgmap v1016: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:24:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:30 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]: dispatch 2024-03-04T16:24:31.141 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:31.201 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:31.238 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:31.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:31.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:30 smithi038 ceph-mon[29779]: pgmap v1016: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:24:31.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:30 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]: dispatch 2024-03-04T16:24:31.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:30 smithi038 ceph-mon[38113]: pgmap v1016: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-04T16:24:31.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:30 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]: dispatch 2024-03-04T16:24:31.340 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:31.379 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:31.481 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:31.524 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:31.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:31.663 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:31.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:31.792 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:31.893 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:31.924 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:31.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:32.025 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:32.057 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:32.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:32.198 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:32.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:31 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]': finished 2024-03-04T16:24:32.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:31 smithi038 ceph-mon[29779]: osdmap e460: 8 total, 8 up, 7 in 2024-03-04T16:24:32.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:31 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]': finished 2024-03-04T16:24:32.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:31 smithi038 ceph-mon[38113]: osdmap e460: 8 total, 8 up, 7 in 2024-03-04T16:24:32.295 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:31 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2057020354' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.6", "id": [5, 7, 2, 1, 6, 0]}]': finished 2024-03-04T16:24:32.295 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:31 smithi176 ceph-mon[33619]: osdmap e460: 8 total, 8 up, 7 in 2024-03-04T16:24:32.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:32.340 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:32.442 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:32.475 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:32.581 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:32.620 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:32.722 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:32.763 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:32.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:32.898 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:32.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:32.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:32.999 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:33.034 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:33.136 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:33.166 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:33.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:33.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:32 smithi038 ceph-mon[29779]: pgmap v1018: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:33.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:32 smithi038 ceph-mon[29779]: osdmap e461: 8 total, 8 up, 7 in 2024-03-04T16:24:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:32 smithi038 ceph-mon[38113]: pgmap v1018: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:32 smithi038 ceph-mon[38113]: osdmap e461: 8 total, 8 up, 7 in 2024-03-04T16:24:33.268 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:33.295 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:33.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:32 smithi176 ceph-mon[33619]: pgmap v1018: 148 pgs: 148 active+clean; 1.1 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:33.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:32 smithi176 ceph-mon[33619]: osdmap e461: 8 total, 8 up, 7 in 2024-03-04T16:24:33.397 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:33.433 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:33.535 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:33.566 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:33.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:33.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:33.706 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:33.742 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:33.808 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:33.843 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:33.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:33.945 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:33.978 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:33.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:34.080 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.122 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:34.223 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:34.271 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:34.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.419 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:34.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:34.565 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:34.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:34.707 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:34.723 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:24:34.755 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:24:34.797 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:24:34.809 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:34.841 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:24:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.850 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:34.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:34.883 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:24:34.920 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:24:34.921 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is set 2024-03-04T16:24:34.951 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:34.962 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:24:34.980 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:34.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:34.990 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[29779]: pgmap v1020: 148 pgs: 148 active+clean; 874 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1272784142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[29779]: from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4106291756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[38113]: pgmap v1020: 148 pgs: 148 active+clean; 874 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1272784142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[38113]: from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:34.991 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4106291756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:24:35.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:24:35.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:24:35.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:24:35.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:24:35.010 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:24:35.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:24:35.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:24:35.011 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:24:35.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.126 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:35.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:35.211 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:35.212 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:34 smithi176 ceph-mon[33619]: pgmap v1020: 148 pgs: 148 active+clean; 874 MiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-04T16:24:35.212 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1272784142' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:35.212 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:34 smithi176 ceph-mon[33619]: from='client.25951 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:35.212 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4106291756' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:35.226 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:35.257 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:35.359 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:35.399 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:35.501 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:35.525 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:35.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.665 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:35.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:35.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:35.804 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:35.895 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:35.896 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:35.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:35.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:36.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:36.093 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:36.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:35 smithi176 ceph-mon[33619]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:36.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:35 smithi176 ceph-mon[33619]: from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:36.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:35 smithi176 ceph-mon[33619]: osdmap e462: 8 total, 8 up, 7 in 2024-03-04T16:24:36.195 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:36.227 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:36.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[29779]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:36.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[29779]: from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:36.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[29779]: osdmap e462: 8 total, 8 up, 7 in 2024-03-04T16:24:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[38113]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[38113]: from='client.25951 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:35 smithi038 ceph-mon[38113]: osdmap e462: 8 total, 8 up, 7 in 2024-03-04T16:24:36.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:36.367 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:36.469 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.513 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:36.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:36.614 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:36.660 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:36.762 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:36.901 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:36.930 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:36.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:36.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:37.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:37.061 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:37.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:37.163 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:37.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:36 smithi038 ceph-mon[29779]: pgmap v1022: 148 pgs: 148 active+clean; 747 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 909 op/s 2024-03-04T16:24:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:36 smithi038 ceph-mon[38113]: pgmap v1022: 148 pgs: 148 active+clean; 747 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 909 op/s 2024-03-04T16:24:37.293 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.324 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:37.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:37.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:36 smithi176 ceph-mon[33619]: pgmap v1022: 148 pgs: 148 active+clean; 747 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 909 op/s 2024-03-04T16:24:37.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:37.469 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:37.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:37.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:37.599 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:37.701 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:37.737 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:37.834 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:24:37.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:37.866 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:37.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:37.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:38.010 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:38.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:38.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.146 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:38.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:38.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:38.285 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:38.290 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:38.359 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:24:38.381 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 7, 2, 6] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:24:38.381 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:24:38.381 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 3, 5, 7, 2, 6] 2024-03-04T16:24:38.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd out 3 2024-03-04T16:24:38.392 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:38.435 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:38.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:38.594 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:38.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:38.677 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:38.696 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:38.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:38.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-04T16:24:38.854 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:38.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:38.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:38.987 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:38.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:38 smithi038 ceph-mon[29779]: pgmap v1023: 148 pgs: 148 active+clean; 659 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 991 op/s 2024-03-04T16:24:38.996 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:38.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:38 smithi038 ceph-mon[38113]: pgmap v1023: 148 pgs: 148 active+clean; 659 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 991 op/s 2024-03-04T16:24:38.997 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:39.031 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:39.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:38 smithi176 ceph-mon[33619]: pgmap v1023: 148 pgs: 148 active+clean; 659 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 740 B/s rd, 0 B/s wr, 991 op/s 2024-03-04T16:24:39.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:39.133 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.184 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:39.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:39.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:39.330 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:39.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:39.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:39.471 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:39.575 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:39.612 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:39.714 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:39.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:39.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:39.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:39.880 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:39.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:39.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:40.010 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:40.066 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:40.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:40.138 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:40.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:40.280 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:40.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:40.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.472 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:40.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:40.573 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:40.601 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:40.703 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:40.746 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:40.821 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:40.848 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:40.881 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:40.982 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:40.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:40 smithi038 ceph-mon[29779]: pgmap v1024: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:24:40.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:40 smithi038 ceph-mon[38113]: pgmap v1024: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:24:40.996 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:40.996 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:40.996 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:40 smithi176 ceph-mon[33619]: pgmap v1024: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-03-04T16:24:41.017 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:41.119 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.159 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:41.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:41.260 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:41.287 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:41.389 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:41.421 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:41.523 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:41.554 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:41.680 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:41.702 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:24:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.719 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:41.820 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:41.844 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:41.946 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:41.968 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:24:41.994 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:41.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:42.029 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:42.096 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:42.131 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:42.233 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:42.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/199383145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:41 smithi038 ceph-mon[29779]: from='client.36035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/199383145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:41 smithi038 ceph-mon[38113]: from='client.36035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.259 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:42.305 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/199383145' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.306 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:41 smithi176 ceph-mon[33619]: from='client.36035 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:24:42.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.405 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:42.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:42.506 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:42.534 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:42.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.679 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:42.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:42.780 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.809 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:42.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:42.910 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:42.941 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:42.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:43.042 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:43.070 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:43.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:43.197 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: pgmap v1025: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:43.197 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:43.197 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: from='client.36035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:24:43.198 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: osdmap e463: 8 total, 8 up, 7 in 2024-03-04T16:24:43.198 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:24:43.198 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:42 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:24:43.207 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:43.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: pgmap v1025: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:43.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:43.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: from='client.36035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: osdmap e463: 8 total, 8 up, 7 in 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: pgmap v1025: 148 pgs: 148 active+clean; 405 MiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: from='client.36035 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: osdmap e463: 8 total, 8 up, 7 in 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:24:43.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:42 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:24:43.309 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:43.337 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:43.439 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:43.473 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:43.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:43.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:43.720 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:43.742 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:43.745 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:43.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:43.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:43.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:43.893 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:43.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:43.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:44.018 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:44.120 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:44.150 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:44.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:44.157 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:43 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:24:44.157 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:43 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:24:44.157 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:43 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:24:44.157 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:43 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:24:44.193 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:44.193 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:44.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:24:44.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:24:44.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:24:44.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:24:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:24:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:24:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:24:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:43 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:24:44.251 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:44.288 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:44.390 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:44.414 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Removed 107136 objects 2024-03-04T16:24:44.415 INFO:tasks.radosbench.radosbench.0.smithi038.stdout:Clean up completed and total clean up time :97.5143 2024-03-04T16:24:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.417 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:44.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:44.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:44.541 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:44.642 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:44.667 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:44.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:44.792 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:44.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:44.917 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:44.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:44.963 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:24:45.019 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:45.042 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:45.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:45.166 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:45.201 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[29779]: pgmap v1027: 148 pgs: 148 active+clean; 184 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:24:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[29779]: 11.b scrub starts 2024-03-04T16:24:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[29779]: 11.b scrub ok 2024-03-04T16:24:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[38113]: pgmap v1027: 148 pgs: 148 active+clean; 184 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:24:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[38113]: 11.b scrub starts 2024-03-04T16:24:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:44 smithi038 ceph-mon[38113]: 11.b scrub ok 2024-03-04T16:24:45.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:45.291 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:45.324 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:45.324 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:44 smithi176 ceph-mon[33619]: pgmap v1027: 148 pgs: 148 active+clean; 184 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-04T16:24:45.324 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:44 smithi176 ceph-mon[33619]: 11.b scrub starts 2024-03-04T16:24:45.324 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:44 smithi176 ceph-mon[33619]: 11.b scrub ok 2024-03-04T16:24:45.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:45.416 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:45.518 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:45.541 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:45.643 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:45.666 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:45.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:45.791 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:45.846 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-04T16:24:45.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-04T16:24:45.893 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:45.916 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:45.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:46.018 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.044 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:46.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:46.145 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.169 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:46.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:46.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:24:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:45 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:24:46.260 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:45 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:24:46.260 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:45 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:24:46.260 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:45 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:24:46.260 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:45 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:24:46.270 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:46.294 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:46.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:46.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:46.419 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:46.520 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:46.544 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:46.645 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:46.669 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:46.752 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:24:46.752 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:24:46.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.795 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:46.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:46.896 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:46.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:46.922 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:46.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:47.024 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.048 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:47.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:47.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.174 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:47.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: pgmap v1028: 148 pgs: 148 active+clean; 62 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 11.0 scrub starts 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 11.0 scrub ok 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:24:47.209 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: pgmap v1028: 148 pgs: 148 active+clean; 62 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 11.0 scrub starts 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 11.0 scrub ok 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:24:47.210 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:46 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:24:47.276 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:47.300 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: pgmap v1028: 148 pgs: 148 active+clean; 62 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 11.0 scrub starts 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 11.0 scrub ok 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:24:47.348 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:24:47.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:24:47.349 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:46 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:24:47.402 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:47.427 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:47.445 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:24:47.529 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:47.558 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:47.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.684 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:47.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:47.725 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:24:47.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:47.810 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:47.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:47 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:24:47.696+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e464 definitely_dead 0 2024-03-04T16:24:47.913 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:47.937 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:47.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:48.038 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:48.059 INFO:teuthology.orchestra.run.smithi038.stderr:marked out osd.3. 2024-03-04T16:24:48.067 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:48.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:24:48.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: osdmap e464: 8 total, 8 up, 7 in 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/668417890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: from='client.36041 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[29779]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-04T16:24:48.097 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: osdmap e464: 8 total, 8 up, 7 in 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/668417890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: from='client.36041 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.098 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:48 smithi038 ceph-mon[38113]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: osdmap e464: 8 total, 8 up, 7 in 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/668417890' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: from='client.36041 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-04T16:24:48.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:48 smithi176 ceph-mon[33619]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-04T16:24:48.134 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:24:48.154 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:24:48.169 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:48.174 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:24:48.193 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:48.194 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:24:48.214 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:24:48.234 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:24:48.255 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:24:48.276 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:24:48.277 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:24:48.295 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:48.318 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:48.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:48.443 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:48.503 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:24:48.545 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:48.570 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:48.672 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:48.700 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:48.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:48.802 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:48.825 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:48.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:48.954 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:48.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:49.055 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.105 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:49.210 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:49.223 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:49.233 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: pgmap v1030: 148 pgs: 148 active+clean; 35 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s 2024-03-04T16:24:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-04T16:24:49.245 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: osdmap e465: 8 total, 8 up, 6 in 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2561050314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: pgmap v1030: 148 pgs: 148 active+clean; 35 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: osdmap e465: 8 total, 8 up, 6 in 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2561050314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:24:49.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:24:49.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:49 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: pgmap v1030: 148 pgs: 148 active+clean; 35 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 740 op/s 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: from='client.36041 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: osdmap e465: 8 total, 8 up, 6 in 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2561050314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:24:49.330 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:49 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:24:49.336 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:49.462 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:49.486 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:49.588 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:49.611 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:49.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:49.740 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "size": 20, 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 600, 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "ops": [ 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223783 10.1 10:81d9e410:::benchmark_data_smithi038_155391_object87826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 283.23989760699999, 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.5165916839999998, 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.741 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223783 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499134+0000", 2024-03-04T16:24:49.742 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499136+0000", 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.159e-06 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499144+0000", 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.1720000000000001e-06 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.743 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499157+0000", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.376e-05 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499168+0000", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0419e-05 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499759+0000", 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00059170200000000005 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.744 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.499771+0000", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1362999999999999e-05 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457342+0000", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95757115599999998 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457355+0000", 2024-03-04T16:24:49.745 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2853999999999999e-05 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693012+0000", 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2356567140000001 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693143+0000", 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000131736 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.746 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.693541+0000", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00039770799999999998 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695555+0000", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020144350000000002 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.709538+0000", 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.013982938 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.747 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.781590+0000", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072051207000000006 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015577+0000", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.233987377 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015635+0000", 2024-03-04T16:24:49.748 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7565000000000002e-05 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.015726+0000", 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1419e-05 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223787 10.10 10:0e9e46b3:::benchmark_data_smithi038_155391_object87830:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:24:49.749 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 283.23501372499999, 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.2362359610000002, 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223787 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:24:49.750 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504018+0000", 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504019+0000", 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1850000000000001e-06 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.751 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504024+0000", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9359999999999998e-06 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504034+0000", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2569999999999996e-06 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504040+0000", 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6619999999999999e-06 2024-03-04T16:24:49.752 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504078+0000", 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8262000000000001e-05 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.504086+0000", 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.199e-06 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.753 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457609+0000", 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95352290900000003 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457620+0000", 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1059e-05 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.575586+0000", 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.11796674 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.754 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.575688+0000", 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010215 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,4,5", 2024-03-04T16:24:49.755 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.576009+0000", 2024-03-04T16:24:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.756 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000320515 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.577642+0000", 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001632582 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.577832+0000", 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00019084700000000001 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.757 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.585829+0000", 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0079967200000000006 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740059+0000", 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.15422976999999999 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740124+0000", 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.5452999999999998e-05 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.758 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.740254+0000", 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129715 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:223796 10.1 10:811c42ae:::benchmark_data_smithi038_155391_object87839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 283.217682024, 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.500384639, 2024-03-04T16:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 223796 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.760 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521350+0000", 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521352+0000", 2024-03-04T16:24:49.761 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7e-06 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521356+0000", 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9089999999999999e-06 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521367+0000", 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0553999999999999e-05 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.762 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521375+0000", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.9359999999999999e-06 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521424+0000", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9509000000000003e-05 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:06.521432+0000", 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8709999999999997e-06 2024-03-04T16:24:49.763 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457381+0000", 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.93594915300000003 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:07.457390+0000", 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.6470000000000004e-06 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.764 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695587+0000", 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.238196968 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695704+0000", 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000116767 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1,4", 2024-03-04T16:24:49.765 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.695954+0000", 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000250344 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.697420+0000", 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0014657100000000001 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.709570+0000", 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.012149927 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.766 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:08.781633+0000", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.072062899999999999 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021608+0000", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.239974826 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021646+0000", 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.8849e-05 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.767 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:09.021734+0000", 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8069e-05 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226044 10.10 10:0f7e88de:::benchmark_data_smithi038_155391_object90087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 277.193629122, 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.015020867, 2024-03-04T16:24:49.768 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226044 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.769 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545403+0000", 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545405+0000", 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9800000000000001e-06 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545410+0000", 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6520000000000003e-06 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.770 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545423+0000", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2283000000000001e-05 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545430+0000", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.5959999999999998e-06 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545484+0000", 2024-03-04T16:24:49.771 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4190000000000001e-05 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.545495+0000", 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0893000000000001e-05 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.553734+0000", 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0082391690000001 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.772 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.553864+0000", 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000129911 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.554229+0000", 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00036436599999999997 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.773 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560197+0000", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0059684860000000003 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560250+0000", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.2518000000000001e-05 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560311+0000", 2024-03-04T16:24:49.774 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1168999999999997e-05 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560367+0000", 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6440999999999997e-05 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560424+0000", 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6212999999999999e-05 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.775 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226056 10.10 10:0bb15e44:::benchmark_data_smithi038_155391_object90099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 277.17461519199998, 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.003719327, 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226056 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.776 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564417+0000", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564418+0000", 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.7150000000000001e-06 2024-03-04T16:24:49.777 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564423+0000", 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5059999999999997e-06 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564432+0000", 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.1579999999999994e-06 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.778 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564437+0000", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3709999999999999e-06 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564494+0000", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6125999999999998e-05 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.564503+0000", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2720000000000003e-06 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.779 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556397+0000", 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99189406999999996 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556524+0000", 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012677000000000001 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.556761+0000", 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000237197 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.780 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560524+0000", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0037634790000000001 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560573+0000", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.8464000000000002e-05 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568028+0000", 2024-03-04T16:24:49.781 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074555200000000002 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568086+0000", 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8031000000000001e-05 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.568136+0000", 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9648000000000003e-05 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.782 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:226060 10.10 10:0d44b17f:::benchmark_data_smithi038_155391_object90103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 277.172026045, 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0032575459999999, 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 226060 2024-03-04T16:24:49.783 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567006+0000", 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.784 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567007+0000", 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.4050000000000001e-06 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567011+0000", 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.245e-06 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567021+0000", 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5559999999999996e-06 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.785 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567026+0000", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.3879999999999999e-06 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567081+0000", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.4713999999999998e-05 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:12.567090+0000", 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.755e-06 2024-03-04T16:24:49.786 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558313+0000", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99122295500000002 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558387+0000", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4598999999999995e-05 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,5", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.558605+0000", 2024-03-04T16:24:49.787 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000218035 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.560630+0000", 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.002024292 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.566064+0000", 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054337939999999996 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.788 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570179+0000", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0041149940000000003 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570221+0000", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2447000000000002e-05 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:13.570263+0000", 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2367000000000003e-05 2024-03-04T16:24:49.789 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230566 10.1 10:85b5cd01:::benchmark_data_smithi038_155391_object94609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 270.30344679299998, 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.039178022, 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230566 2024-03-04T16:24:49.790 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435585+0000", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435586+0000", 2024-03-04T16:24:49.791 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.378e-06 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435591+0000", 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1919999999999998e-06 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435597+0000", 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8050000000000001e-06 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.792 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435603+0000", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1000000000000003e-06 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435671+0000", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8094000000000002e-05 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.435683+0000", 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2036e-05 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.793 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465007+0000", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0293241689999999 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465098+0000", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.0810999999999995e-05 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.465450+0000", 2024-03-04T16:24:49.794 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00035259899999999999 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.470910+0000", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0054596419999999998 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.470990+0000", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0610000000000002e-05 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474680+0000", 2024-03-04T16:24:49.795 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0036890199999999999 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474735+0000", 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5658999999999997e-05 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474763+0000", 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.7906999999999999e-05 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.796 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230589 10.1 10:80d77cf6:::benchmark_data_smithi038_155391_object94632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 270.26758273799999, 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0184885720000001, 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230589 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.797 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471449+0000", 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.798 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471451+0000", 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9939999999999999e-06 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471457+0000", 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6110000000000001e-06 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471469+0000", 2024-03-04T16:24:49.799 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2109999999999999e-05 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471475+0000", 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.331e-06 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471545+0000", 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9733999999999993e-05 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.800 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.471555+0000", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7999999999999993e-06 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467374+0000", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99581904700000001 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467451+0000", 2024-03-04T16:24:49.801 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.7729000000000001e-05 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.467611+0000", 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015940400000000001 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472118+0000", 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0045073149999999996 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.802 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479527+0000", 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074085660000000001 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489845+0000", 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010318539999999999 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489895+0000", 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0170999999999998e-05 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.803 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489938+0000", 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.2219999999999999e-05 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230616 10.1 10:858a950a:::benchmark_data_smithi038_155391_object94659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 270.22492390500003, 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.97592274199999995, 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.804 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230616 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.805 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514108+0000", 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514110+0000", 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.652e-06 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514115+0000", 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0769999999999998e-06 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.806 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514125+0000", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9529999999999994e-06 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514130+0000", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5269999999999998e-06 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514176+0000", 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.6238e-05 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.807 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.514182+0000", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0299999999999999e-06 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468424+0000", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95424162899999998 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468529+0000", 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000104589 2024-03-04T16:24:49.808 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.468781+0000", 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00025258799999999999 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.472121+0000", 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0033400690000000002 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.809 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479568+0000", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074465130000000001 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.489972+0000", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010404681000000001 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490006+0000", 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.3556999999999997e-05 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.810 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490031+0000", 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.4638999999999999e-05 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.35585.0:230624 10.1 10:8414195d:::benchmark_data_smithi038_155391_object94667:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 270.21330531500001, 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96442188399999995, 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.811 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.35585", 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/361269024", 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 230624 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.812 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525727+0000", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525729+0000", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0169999999999999e-06 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525735+0000", 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0939999999999996e-06 2024-03-04T16:24:49.813 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.814 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.814 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.814 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525745+0000", 2024-03-04T16:24:49.814 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:49.815 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:49.816 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0638e-05 2024-03-04T16:24:49.816 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525752+0000", 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.199e-06 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525802+0000", 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9988000000000003e-05 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.817 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:19.525809+0000", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.1700000000000002e-06 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469771+0000", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.94396249899999995 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469840+0000", 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.8458000000000001e-05 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.818 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 0,1", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.469999+0000", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000159801 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.474635+0000", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0046351999999999999 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.479605+0000", 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0049702929999999998 2024-03-04T16:24:49.819 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490099+0000", 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010494111 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490126+0000", 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6585999999999999e-05 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.820 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:20:20.490148+0000", 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.283e-05 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:16722 11.5 11:a0d27429:::benchmark_data_smithi038_182448_object16721:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:44.545927+0000", 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.19310437499999, 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.974694516, 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:24:49.821 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 16722 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545927+0000", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545927+0000", 2024-03-04T16:24:49.822 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545929+0000", 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3030000000000001e-06 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545969+0000", 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.0030000000000001e-05 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.823 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545989+0000", 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0361e-05 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.545997+0000", 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8760000000000005e-06 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546057+0000", 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.9715e-05 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.824 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546224+0000", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016771200000000001 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547127+0000", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00090250499999999995 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.554579+0000", 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0074519110000000003 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.825 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556406+0000", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0018275940000000001 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556420+0000", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3735e-05 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404332+0000", 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84791129300000001 2024-03-04T16:24:49.826 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404350+0000", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.8255999999999999e-05 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.512737+0000", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1083869019999999 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.513479+0000", 2024-03-04T16:24:49.827 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00074222000000000003 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.520506+0000", 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0070266130000000001 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.520622+0000", 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011649 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.828 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:16723 11.5 11:a6e3d04e:::benchmark_data_smithi038_182448_object16722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:44.546018+0000", 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.19301336500001, 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.976442984, 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 16723 2024-03-04T16:24:49.829 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546018+0000", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546018+0000", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546019+0000", 2024-03-04T16:24:49.830 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.2099999999999995e-07 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546056+0000", 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6749999999999999e-05 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546062+0000", 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0399999999999998e-06 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.831 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546068+0000", 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8880000000000002e-06 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.553396+0000", 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0073282319999999996 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.832 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.553545+0000", 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014831299999999999 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 3,5", 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.553693+0000", 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014812899999999999 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556454+0000", 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0027615980000000001 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.833 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556464+0000", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0020000000000001e-05 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404391+0000", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84792665899999997 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404401+0000", 2024-03-04T16:24:49.834 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0315e-05 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.514239+0000", 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.109837223 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for degraded object", 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.516239+0000", 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020006870000000001 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.835 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.522318+0000", 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0060785910000000004 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.522461+0000", 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014361800000000001 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.836 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:16727 11.5 11:ab71b5aa:::benchmark_data_smithi038_182448_object16726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:44.546077+0000", 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.19295521699999, 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9769387250000001, 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 16727 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.837 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546077+0000", 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546077+0000", 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.838 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546077+0000", 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.0200000000000001e-07 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546106+0000", 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.8581e-05 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546112+0000", 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5799999999999999e-06 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.839 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.546117+0000", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8590000000000001e-06 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556527+0000", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.010409186000000001 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.556532+0000", 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.271e-06 2024-03-04T16:24:49.840 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404546+0000", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84801454799999998 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.404555+0000", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8049999999999996e-06 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.516323+0000", 2024-03-04T16:24:49.841 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.111768036 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.516483+0000", 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015998799999999999 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 5,6", 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.516697+0000", 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000213804 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.842 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.520704+0000", 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0040067189999999997 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.521338+0000", 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00063400900000000005 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.522501+0000", 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00116333 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.843 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.522576+0000", 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4598000000000006e-05 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.523015+0000", 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00043960899999999999 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.844 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:16730 11.4 11:29637677:::benchmark_data_smithi038_182448_object16729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:44.547378+0000", 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.19165354200001, 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.964340768, 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 16730 2024-03-04T16:24:49.845 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547378+0000", 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547378+0000", 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.846 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547380+0000", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.421e-06 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547416+0000", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 3.6220000000000002e-05 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547424+0000", 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.259e-06 2024-03-04T16:24:49.847 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.547434+0000", 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3190000000000001e-06 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.558935+0000", 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.011501503 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.848 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.558945+0000", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.9150000000000008e-06 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.406964+0000", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.84801859499999999 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.406974+0000", 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0197000000000001e-05 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.849 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.503398+0000", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0964242719999999 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.503555+0000", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015655399999999999 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.503679+0000", 2024-03-04T16:24:49.850 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012475 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.506847+0000", 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0031677739999999999 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.506914+0000", 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.7088000000000005e-05 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.851 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.510870+0000", 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.003955704 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.510936+0000", 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6104000000000003e-05 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.511719+0000", 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00078309300000000001 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.852 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:16734 11.4 11:28643f51:::benchmark_data_smithi038_182448_object16733:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:44.574456+0000", 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 185.16457632800001, 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.937885471, 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.853 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 16734 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574456+0000", 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574456+0000", 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.854 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574458+0000", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.6000000000000001e-06 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574503+0000", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5284999999999998e-05 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574519+0000", 2024-03-04T16:24:49.855 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5624000000000002e-05 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574530+0000", 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0526999999999999e-05 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574596+0000", 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6004e-05 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.856 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:44.574607+0000", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1745e-05 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.407279+0000", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.83267159700000004 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:45.407300+0000", 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.0907000000000002e-05 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.857 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.505581+0000", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.09828131 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.505757+0000", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017600100000000001 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.505954+0000", 2024-03-04T16:24:49.858 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000197252 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.506901+0000", 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00094649999999999997 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.507013+0000", 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000112056 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.859 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.511784+0000", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.004771216 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.511850+0000", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.6148000000000004e-05 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:46.512341+0000", 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00049069900000000004 2024-03-04T16:24:49.860 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:24174 11.4 11:283c5ed7:::benchmark_data_smithi038_182448_object24173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:51.414829+0000", 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 178.324202754, 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0128556500000001, 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.861 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24174 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414829+0000", 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414829+0000", 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.862 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414831+0000", 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.9209999999999999e-06 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414889+0000", 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.7614000000000001e-05 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414901+0000", 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.2357999999999999e-05 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.863 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.414909+0000", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.8909999999999995e-06 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.415193+0000", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00028410700000000002 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.415354+0000", 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00016075400000000001 2024-03-04T16:24:49.864 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.415480+0000", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012588899999999999 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.417484+0000", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0020040460000000002 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.417634+0000", 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00015060099999999999 2024-03-04T16:24:49.865 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.422931+0000", 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052972150000000001 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.422945+0000", 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3816e-05 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.866 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.427633+0000", 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0046880010000001 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.427685+0000", 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.1437000000000002e-05 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:24179 11.4 11:20362673:::benchmark_data_smithi038_182448_object24178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-04T16:24:49.867 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:51.416150+0000", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 178.32288178499999, 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.011917285, 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "reached pg", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24179 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416150+0000", 2024-03-04T16:24:49.868 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416150+0000", 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416152+0000", 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.5519999999999999e-06 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.869 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416226+0000", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.4268000000000001e-05 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416238+0000", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1747e-05 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416246+0000", 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.1610000000000003e-06 2024-03-04T16:24:49.870 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416343+0000", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.7081000000000001e-05 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416514+0000", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00017157899999999999 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.416695+0000", 2024-03-04T16:24:49.871 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00018005699999999999 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.417659+0000", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00096480700000000003 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.417808+0000", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00014830799999999999 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423043+0000", 2024-03-04T16:24:49.872 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0052348760000000003 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423053+0000", 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0492000000000001e-05 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.428026+0000", 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0049726640000001 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.873 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.428067+0000", 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.1693000000000002e-05 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:24183 11.4 11:28f02527:::benchmark_data_smithi038_182448_object24182:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:51.423628+0000", 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 178.31540405699999, 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0119690079999999, 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.874 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24183 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423628+0000", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423628+0000", 2024-03-04T16:24:49.875 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423630+0000", 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.1399999999999998e-06 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423680+0000", 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.9806999999999999e-05 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.876 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423690+0000", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.042e-05 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423698+0000", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.14e-06 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423753+0000", 2024-03-04T16:24:49.877 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.5157000000000001e-05 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.423763+0000", 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.3149999999999998e-06 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.428104+0000", 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.0043408309999999 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.878 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.428232+0000", 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000128796 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.428342+0000", 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00010962 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.879 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.430827+0000", 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0024851209999999999 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.431072+0000", 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000245262 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435503+0000", 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0044303379999999998 2024-03-04T16:24:49.880 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435553+0000", 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0127000000000003e-05 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435597+0000", 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.3934000000000003e-05 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.881 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:24196 11.4 11:221b6ddc:::benchmark_data_smithi038_182448_object24195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:51.445388+0000", 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 178.293643715, 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.99038243199999998, 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24196 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.882 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445388+0000", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445388+0000", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445389+0000", 2024-03-04T16:24:49.883 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.3030000000000001e-06 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445478+0000", 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.8221999999999997e-05 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445487+0000", 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 9.5650000000000007e-06 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.884 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445496+0000", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 8.4740000000000005e-06 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445546+0000", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.0497000000000001e-05 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.445557+0000", 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1242e-05 2024-03-04T16:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.429070+0000", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.98351284100000003 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.429196+0000", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00012574400000000001 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.429316+0000", 2024-03-04T16:24:49.886 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011989099999999999 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.430891+0000", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0015746149999999999 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.432209+0000", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.001318201 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435651+0000", 2024-03-04T16:24:49.887 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0034425570000000002 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435725+0000", 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 7.3872e-05 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435771+0000", 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.5408000000000003e-05 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.888 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "description": "osd_op(client.16017.0:24222 11.4 11:28fb5d7f:::benchmark_data_smithi038_182448_object24221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "initiated_at": "2024-03-04T16:21:51.470486+0000", 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "age": 178.268545956, 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.96544412499999999, 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "type_data": { 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "client_info": { 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "client": "client.16017", 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "client_addr": "v1:172.21.15.38:0/3130059731", 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "tid": 24222 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.889 INFO:teuthology.orchestra.run.smithi176.stdout: "events": [ 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "initiated", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470486+0000", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "header_read", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470486+0000", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "throttled", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470488+0000", 2024-03-04T16:24:49.890 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 2.3e-06 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "all_read", 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470547+0000", 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8743000000000003e-05 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "dispatched", 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470559+0000", 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 1.1861e-05 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "queued_for_pg", 2024-03-04T16:24:49.891 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470565+0000", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.8420000000000001e-06 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470625+0000", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.0007999999999999e-05 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for peered", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:51.470632+0000", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 6.9489999999999999e-06 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "reached_pg", 2024-03-04T16:24:49.892 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.429973+0000", 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.95934114699999995 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "started", 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.430090+0000", 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.00011695400000000001 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "waiting for subops from 2,6", 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.430226+0000", 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.000136231 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.893 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "op_commit", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.432215+0000", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0019886859999999999 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.433469+0000", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0012540030000000001 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "sub_op_commit_rec", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435827+0000", 2024-03-04T16:24:49.894 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 0.0023579109999999999 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "commit_sent", 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435874+0000", 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 4.7157999999999998e-05 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: }, 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: { 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "event": "done", 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "time": "2024-03-04T16:21:52.435930+0000", 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: "duration": 5.6332000000000002e-05 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.895 INFO:teuthology.orchestra.run.smithi176.stdout: } 2024-03-04T16:24:49.896 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-03-04T16:24:49.896 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-03-04T16:24:49.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:49.938 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:49.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:50.040 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:50.071 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:50.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:50.199 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:50.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:50.301 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:50.304 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: osdmap e466: 8 total, 8 up, 6 in 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[29779]: osdmap e467: 8 total, 8 up, 6 in 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:24:50.305 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: osdmap e466: 8 total, 8 up, 6 in 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:50.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:50 smithi038 ceph-mon[38113]: osdmap e467: 8 total, 8 up, 6 in 2024-03-04T16:24:50.326 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:50.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:50.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: osdmap e466: 8 total, 8 up, 6 in 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:50 smithi176 ceph-mon[33619]: osdmap e467: 8 total, 8 up, 6 in 2024-03-04T16:24:50.428 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.455 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:50.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:50.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:50.579 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:50.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:50.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:50.704 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:50.706 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-04T16:24:50.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:50.830 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:50.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:50.957 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:50.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:51.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:51.082 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:51.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:51.208 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:51.310 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:51.334 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:51.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: pgmap v1033: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 489 op/s; 213 B/s, 8 objects/s recovering 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 463'13614 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 463'13536 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 463'13408 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-04T16:24:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:51 smithi176 ceph-mon[33619]: osdmap e468: 8 total, 8 up, 6 in 2024-03-04T16:24:51.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: pgmap v1033: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 489 op/s; 213 B/s, 8 objects/s recovering 2024-03-04T16:24:51.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 463'13614 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 463'13536 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 463'13408 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[29779]: osdmap e468: 8 total, 8 up, 6 in 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: pgmap v1033: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 B/s wr, 489 op/s; 213 B/s, 8 objects/s recovering 2024-03-04T16:24:51.379 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.2 starting backfill to osd.7 from (0'0,0'0] MAX to 463'13614 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.9 starting backfill to osd.1 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.8 starting backfill to osd.0 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.1 starting backfill to osd.2 from (0'0,0'0] MAX to 463'13536 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.a starting backfill to osd.1 from (0'0,0'0] MAX to 463'13280 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.0 starting backfill to osd.6 from (0'0,0'0] MAX to 463'13252 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 463'13408 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-04T16:24:51.380 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:51 smithi038 ceph-mon[38113]: osdmap e468: 8 total, 8 up, 6 in 2024-03-04T16:24:51.437 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.461 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:51.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:51.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:51.586 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:51.688 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:51.713 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:51.814 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:51.816 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-04T16:24:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.843 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:51.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:51.947 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:51.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:51.971 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:51.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:52.072 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:52.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:24:52.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:24:52.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[29779]: osdmap e469: 8 total, 8 up, 6 in 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:24:52.092 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:52 smithi038 ceph-mon[38113]: osdmap e469: 8 total, 8 up, 6 in 2024-03-04T16:24:52.096 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:52.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:52.223 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:52.325 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:52.351 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:52.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:52 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:24:52.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:52 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:24:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:52 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:24:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:52 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:24:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:52 smithi176 ceph-mon[33619]: osdmap e469: 8 total, 8 up, 6 in 2024-03-04T16:24:52.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:52.476 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:52.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:52.606 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:52.708 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:52.732 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:52.833 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3 2024-03-04T16:24:52.858 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.3' 2024-03-04T16:24:52.960 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0 2024-03-04T16:24:52.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:52.985 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:52.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.0' 2024-03-04T16:24:53.086 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:53.112 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: 11.f scrub starts 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: 11.f scrub ok 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: pgmap v1037: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 341 B/s, 13 objects/s recovering 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2483303927' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[29779]: from='client.36053 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: 11.f scrub starts 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: 11.f scrub ok 2024-03-04T16:24:53.178 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: pgmap v1037: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 341 B/s, 13 objects/s recovering 2024-03-04T16:24:53.179 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:24:53.179 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:24:53.179 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2483303927' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.179 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-mon[38113]: from='client.36053 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:53.237 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:53.339 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:53.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: 11.f scrub starts 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: 11.f scrub ok 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: pgmap v1037: 148 pgs: 2 remapped+peering, 17 peering, 129 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 64 op/s; 341 B/s, 13 objects/s recovering 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2483303927' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-mon[33619]: from='client.36053 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-04T16:24:53.363 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:53.460 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:53.464 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:53.495 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:24:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:24:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:24:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.498 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:53.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:53.507 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'unique_pool_4' removed 2024-03-04T16:24:53.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.626 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:53.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:53.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2 2024-03-04T16:24:53.752 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.2' 2024-03-04T16:24:53.854 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:53.879 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:53.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:53.953 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 5, 7, 2, 6] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-04T16:24:53.954 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-04T16:24:53.954 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-04T16:24:53.954 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd test-reweight-by-utilization 110 3.0 2024-03-04T16:24:53.980 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7 2024-03-04T16:24:54.004 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.7' 2024-03-04T16:24:54.106 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:54.107 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:24:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:24:54.130 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:54.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:54.255 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:54.357 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5 2024-03-04T16:24:54.380 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.5' 2024-03-04T16:24:54.482 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:54.490 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:54.492 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:24:54.506 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:54.608 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6 2024-03-04T16:24:54.632 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.6' 2024-03-04T16:24:54.734 DEBUG:teuthology.orchestra.run.smithi038:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1 2024-03-04T16:24:54.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:24:54.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: osdmap e470: 8 total, 8 up, 6 in 2024-03-04T16:24:54.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/454293631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: from='client.36059 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: 3.1a scrub starts 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[29779]: 3.1a scrub ok 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: osdmap e470: 8 total, 8 up, 6 in 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:24:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/454293631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: from='client.36059 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: 3.1a scrub starts 2024-03-04T16:24:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:54 smithi038 ceph-mon[38113]: 3.1a scrub ok 2024-03-04T16:24:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.760 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: docker: command not found 2024-03-04T16:24:54.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi038 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.1' 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: from='client.36053 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: osdmap e470: 8 total, 8 up, 6 in 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/454293631' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: from='client.36059 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:24:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: 3.1a scrub starts 2024-03-04T16:24:54.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:54 smithi176 ceph-mon[33619]: 3.1a scrub ok 2024-03-04T16:24:54.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4 2024-03-04T16:24:54.885 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: docker: command not found 2024-03-04T16:24:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:24:54.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi176 with status 1: 'sudo docker kill -s 1 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd.4' 2024-03-04T16:24:54.956 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-04T16:24:54.966 INFO:tasks.thrashosds:joining thrashosds 2024-03-04T16:24:55.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:55.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:55.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: from='client.36059 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: osdmap e471: 8 total, 8 up, 6 in 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:24:55.736 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[29779]: osdmap e472: 8 total, 8 up, 6 in 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:55.737 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: from='client.36059 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: osdmap e471: 8 total, 8 up, 6 in 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:24:55.738 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:55 smithi038 ceph-mon[38113]: osdmap e472: 8 total, 8 up, 6 in 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: pgmap v1039: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: from='client.36059 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: osdmap e471: 8 total, 8 up, 6 in 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:24:55.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:55 smithi176 ceph-mon[33619]: osdmap e472: 8 total, 8 up, 6 in 2024-03-04T16:24:56.212 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:24:56.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[29779]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:24:56.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:24:56.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:56 smithi038 ceph-mon[38113]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:56 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:24:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:56 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:24:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:56 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:24:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:56 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:24:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:56 smithi176 ceph-mon[33619]: pgmap v1042: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:57.794 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:57 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:24:57.794 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:57 smithi038 ceph-mon[29779]: 5.7 scrub ok 2024-03-04T16:24:57.794 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:57 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:24:57.794 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:57 smithi038 ceph-mon[38113]: 5.7 scrub ok 2024-03-04T16:24:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:57 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:24:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:57 smithi176 ceph-mon[33619]: 5.7 scrub ok 2024-03-04T16:24:58.203 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset noscrub 2024-03-04T16:24:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0: {} 2024-03-04T16:24:58.420 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1: {} 2024-03-04T16:24:58.425 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2: {} 2024-03-04T16:24:58.446 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3: {} 2024-03-04T16:24:58.476 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4: {} 2024-03-04T16:24:58.497 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5: {} 2024-03-04T16:24:58.518 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6: {} 2024-03-04T16:24:58.542 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7: {} 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-04T16:24:58.543 INFO:teuthology.orchestra.run.smithi038.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-04T16:24:58.544 INFO:teuthology.orchestra.run.smithi038.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:24:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/306702694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:58.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:24:58.882 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/306702694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: pgmap v1043: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:24:58.883 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/306702694' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[29779]: 3.6 scrub starts 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[29779]: 3.6 scrub ok 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[29779]: osdmap e473: 8 total, 8 up, 6 in 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[38113]: 3.6 scrub starts 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[38113]: 3.6 scrub ok 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:24:59.810 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:24:59.811 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:24:59 smithi038 ceph-mon[38113]: osdmap e473: 8 total, 8 up, 6 in 2024-03-04T16:24:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:59 smithi176 ceph-mon[33619]: 3.6 scrub starts 2024-03-04T16:24:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:59 smithi176 ceph-mon[33619]: 3.6 scrub ok 2024-03-04T16:24:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:59 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:24:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:59 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:24:59.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:24:59 smithi176 ceph-mon[33619]: osdmap e473: 8 total, 8 up, 6 in 2024-03-04T16:25:00.031 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:moved 29 / 396 (7.32323%) 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:avg 66 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 6.13732 -> 5.1316 (expected baseline 7.4162) 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.2 with 56 -> 65 pgs (0.848485 -> 0.984848 * mean) 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 76 -> 57 pgs (1.15152 -> 0.863636 * mean) 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 3 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0091 2024-03-04T16:25:00.267 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0100 2024-03-04T16:25:00.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 1.0000 -> 0.7495 2024-03-04T16:25:00.268 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 weight 1.0000 -> 0.8314 2024-03-04T16:25:00.269 INFO:teuthology.orchestra.run.smithi038.stderr:no change 2024-03-04T16:25:00.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:25:00.265+0000 7fce816b4700 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: osdmap e474: 8 total, 8 up, 6 in 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: 6.d deep-scrub starts 2024-03-04T16:25:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: from='client.25993 v1:172.21.15.38:0/2229380614' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:00.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:00 smithi176 ceph-mon[33619]: osdmap e475: 8 total, 8 up, 6 in 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: osdmap e474: 8 total, 8 up, 6 in 2024-03-04T16:25:00.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: 6.d deep-scrub starts 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: from='client.25993 v1:172.21.15.38:0/2229380614' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[29779]: osdmap e475: 8 total, 8 up, 6 in 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: osdmap e474: 8 total, 8 up, 6 in 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:25:00.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: pgmap v1046: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 967 B/s rd, 0 op/s 2024-03-04T16:25:00.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: 6.d deep-scrub starts 2024-03-04T16:25:00.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: from='client.25993 v1:172.21.15.38:0/2229380614' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:00.996 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:00 smithi038 ceph-mon[38113]: osdmap e475: 8 total, 8 up, 6 in 2024-03-04T16:25:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.d deep-scrub ok 2024-03-04T16:25:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:25:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:25:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 4.7 deep-scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 4.7 deep-scrub ok 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.d deep-scrub ok 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:25:01.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 4.7 deep-scrub starts 2024-03-04T16:25:01.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 4.7 deep-scrub ok 2024-03-04T16:25:01.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:01 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.d deep-scrub ok 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 4.7 deep-scrub starts 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 4.7 deep-scrub ok 2024-03-04T16:25:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:01 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:25:01.869 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-04T16:25:01.869 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-03-04T16:25:02.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:25:02.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:25:02.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:25:02.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:25:02.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[29779]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:25:02.637 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:02 smithi038 ceph-mon[38113]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:02.761 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:02.761 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:25:02.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:25:02.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:25:02.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:25:02.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:25:02.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:02 smithi176 ceph-mon[33619]: pgmap v1048: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:03.517 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:03.536 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-04T16:25:03.560 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:03.590 INFO:teuthology.orchestra.run.smithi038.stderr:noscrub is unset 2024-03-04T16:25:03.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:03 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:25:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:03 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:25:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:03 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1928834367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:03 smithi176 ceph-mon[33619]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:03.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1928834367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[29779]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1928834367' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:03 smithi038 ceph-mon[38113]: from='client.36077 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: 3.12 scrub starts 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: 3.12 scrub ok 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: osdmap e476: 8 total, 8 up, 6 in 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 977 B/s rd, 0 op/s 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:25:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:04 smithi176 ceph-mon[33619]: 6.9 deep-scrub starts 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: 3.12 scrub starts 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: 3.12 scrub ok 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: osdmap e476: 8 total, 8 up, 6 in 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 977 B/s rd, 0 op/s 2024-03-04T16:25:04.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[38113]: 6.9 deep-scrub starts 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: 3.12 scrub starts 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: 3.12 scrub ok 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: from='client.36077 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: osdmap e476: 8 total, 8 up, 6 in 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: pgmap v1050: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 977 B/s rd, 0 op/s 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:25:04.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:04 smithi038 ceph-mon[29779]: 6.9 deep-scrub starts 2024-03-04T16:25:04.894 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-04T16:25:05.372 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:05.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:25:05.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[29779]: 6.9 deep-scrub ok 2024-03-04T16:25:05.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:25:05.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:25:05.735 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[29779]: osdmap e477: 8 total, 8 up, 6 in 2024-03-04T16:25:05.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:25:05.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[38113]: 6.9 deep-scrub ok 2024-03-04T16:25:05.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:25:05.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:25:05.736 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:05 smithi038 ceph-mon[38113]: osdmap e477: 8 total, 8 up, 6 in 2024-03-04T16:25:05.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:05 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:25:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:05 smithi176 ceph-mon[33619]: 6.9 deep-scrub ok 2024-03-04T16:25:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:05 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:25:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:05 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:25:05.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:05 smithi176 ceph-mon[33619]: osdmap e477: 8 total, 8 up, 6 in 2024-03-04T16:25:06.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:06 smithi176 ceph-mon[33619]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:06 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:25:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:06 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[29779]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[38113]: pgmap v1052: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:25:06.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:06 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:25:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:07 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:25:07.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:07 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:25:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:07 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:25:07.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:07 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:25:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:07 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:25:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:07 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:25:08.625 INFO:teuthology.orchestra.run.smithi038.stdout:moved 30 / 396 (7.57576%) 2024-03-04T16:25:08.625 INFO:teuthology.orchestra.run.smithi038.stdout:avg 66 2024-03-04T16:25:08.625 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 6.13732 -> 5.38516 (expected baseline 7.4162) 2024-03-04T16:25:08.625 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.2 with 56 -> 65 pgs (0.848485 -> 0.984848 * mean) 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.1 with 76 -> 56 pgs (1.15152 -> 0.848485 * mean) 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 1 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0091 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0100 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 1.0000 -> 0.7474 2024-03-04T16:25:08.626 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 weight 1.0000 -> 0.8321 2024-03-04T16:25:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:25:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:25:08.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 700 B/s rd, 0 op/s 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 700 B/s rd, 0 op/s 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:08.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:08 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: pgmap v1053: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 700 B/s rd, 0 op/s 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:08 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]: dispatch 2024-03-04T16:25:09.200 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:09.200 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:09.518 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-04T16:25:09.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight-by-utilization 110 0.05 2024-03-04T16:25:09.618 INFO:teuthology.orchestra.run.smithi038.stderr:nodeep-scrub is unset 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: 6.c scrub starts 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: 6.c scrub ok 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='client.36083 v1:172.21.15.38:0/537441331' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]': finished 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: osdmap e478: 8 total, 8 up, 6 in 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:09.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[38113]: osdmap e479: 8 total, 8 up, 6 in 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: 6.c scrub starts 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: 6.c scrub ok 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='client.36083 v1:172.21.15.38:0/537441331' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]': finished 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: osdmap e478: 8 total, 8 up, 6 in 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:09.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:09.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:25:09.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:25:09.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:09 smithi038 ceph-mon[29779]: osdmap e479: 8 total, 8 up, 6 in 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: 6.c scrub starts 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: 6.c scrub ok 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='client.36083 v1:172.21.15.38:0/537441331' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48979','5':'54531'}"}]': finished 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: osdmap e478: 8 total, 8 up, 6 in 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-04T16:25:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3278016810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-04T16:25:10.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:09 smithi176 ceph-mon[33619]: osdmap e479: 8 total, 8 up, 6 in 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 6.1e deep-scrub starts 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 6.1e deep-scrub ok 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 3.0 scrub starts 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: 3.0 scrub ok 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[38113]: pgmap v1056: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 6.1e deep-scrub starts 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 6.1e deep-scrub ok 2024-03-04T16:25:10.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:25:10.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:25:10.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 3.0 scrub starts 2024-03-04T16:25:10.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: 3.0 scrub ok 2024-03-04T16:25:10.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:10 smithi038 ceph-mon[29779]: pgmap v1056: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:10.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-04T16:25:10.911 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 6.1e deep-scrub starts 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 6.1e deep-scrub ok 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 3.0 scrub starts 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: 3.0 scrub ok 2024-03-04T16:25:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:10 smithi176 ceph-mon[33619]: pgmap v1056: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:11.727 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:11.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:25:11.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:25:11.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[29779]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:11.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:25:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:25:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:11 smithi038 ceph-mon[38113]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:11 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:25:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:11 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:25:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:11 smithi176 ceph-mon[33619]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:25:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:25:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:25:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:25:12.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[29779]: pgmap v1057: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:25:12.958 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:25:12.958 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:25:12.958 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:25:12.958 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:25:12.958 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:12 smithi038 ceph-mon[38113]: pgmap v1057: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:25:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:12 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:25:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:12 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:25:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:12 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:25:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:12 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:25:13.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:12 smithi176 ceph-mon[33619]: pgmap v1057: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 811 B/s rd, 0 op/s 2024-03-04T16:25:13.666 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:13.666 INFO:teuthology.orchestra.run.smithi038.stdout:moved 7 / 396 (1.76768%) 2024-03-04T16:25:13.666 INFO:teuthology.orchestra.run.smithi038.stdout:avg 66 2024-03-04T16:25:13.666 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 5.38516 -> 7 (expected baseline 7.4162) 2024-03-04T16:25:13.666 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.1 with 56 -> 54 pgs (0.848485 -> 0.818182 * mean) 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.7 with 74 -> 76 pgs (1.12121 -> 1.15152 * mean) 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 0.05 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0091 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0100 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.7474 -> 0.6974 2024-03-04T16:25:13.667 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 weight 0.8321 -> 0.7821 2024-03-04T16:25:13.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 6.4 deep-scrub starts 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: 6.4 deep-scrub ok 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:25:13.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 6.4 deep-scrub starts 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: 6.4 deep-scrub ok 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:13.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:13 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:14.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:25:14.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 6.4 deep-scrub starts 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: 6.4 deep-scrub ok 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:14.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:13 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]: dispatch 2024-03-04T16:25:14.433 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-04T16:25:14.433 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-03-04T16:25:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: 6.1c deep-scrub starts 2024-03-04T16:25:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: 6.1c deep-scrub ok 2024-03-04T16:25:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: from='client.16269 v1:172.21.15.38:0/829660778' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]': finished 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: osdmap e480: 8 total, 8 up, 6 in 2024-03-04T16:25:14.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:14 smithi176 ceph-mon[33619]: pgmap v1059: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 317 B/s, 3 objects/s recovering 2024-03-04T16:25:14.941 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:14.941 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:14.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: 6.1c deep-scrub starts 2024-03-04T16:25:14.941 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: 6.1c deep-scrub ok 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: from='client.16269 v1:172.21.15.38:0/829660778' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]': finished 2024-03-04T16:25:14.942 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: osdmap e480: 8 total, 8 up, 6 in 2024-03-04T16:25:14.943 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[29779]: pgmap v1059: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 317 B/s, 3 objects/s recovering 2024-03-04T16:25:14.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: 6.1c deep-scrub starts 2024-03-04T16:25:14.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: 6.1c deep-scrub ok 2024-03-04T16:25:14.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: from='client.16269 v1:172.21.15.38:0/829660778' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:14.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:25:14.944 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:25:14.944 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:14.944 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'45703','5':'51255'}"}]': finished 2024-03-04T16:25:14.944 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: osdmap e480: 8 total, 8 up, 6 in 2024-03-04T16:25:14.944 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:14 smithi038 ceph-mon[38113]: pgmap v1059: 132 pgs: 5 peering, 127 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 317 B/s, 3 objects/s recovering 2024-03-04T16:25:15.199 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:15.754 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-04T16:25:15.952 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 4.e scrub starts 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 4.e scrub ok 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: osdmap e481: 8 total, 8 up, 6 in 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 5.b deep-scrub starts 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[38113]: 5.b deep-scrub ok 2024-03-04T16:25:15.953 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 4.e scrub starts 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 4.e scrub ok 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: osdmap e481: 8 total, 8 up, 6 in 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 5.b deep-scrub starts 2024-03-04T16:25:15.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:15 smithi038 ceph-mon[29779]: 5.b deep-scrub ok 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 4.e scrub starts 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 4.e scrub ok 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: osdmap e481: 8 total, 8 up, 6 in 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 5.b deep-scrub starts 2024-03-04T16:25:16.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:15 smithi176 ceph-mon[33619]: 5.b deep-scrub ok 2024-03-04T16:25:16.940 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:16.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:16 smithi038 ceph-mon[29779]: pgmap v1061: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 344 B/s, 3 objects/s recovering 2024-03-04T16:25:16.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:16 smithi038 ceph-mon[38113]: pgmap v1061: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 344 B/s, 3 objects/s recovering 2024-03-04T16:25:17.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:16 smithi176 ceph-mon[33619]: pgmap v1061: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 344 B/s, 3 objects/s recovering 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.e deep-scrub starts 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.e deep-scrub ok 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 4.13 scrub starts 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 4.13 scrub ok 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:25:17.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.e deep-scrub starts 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.e deep-scrub ok 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 4.13 scrub starts 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 4.13 scrub ok 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:25:17.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:17 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.e deep-scrub starts 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.e deep-scrub ok 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 4.13 scrub starts 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 4.13 scrub ok 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: Health check update: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:25:18.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:17 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:25:18.887 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:18.888 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: pgmap v1062: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 265 B/s, 2 objects/s recovering 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:25:18.888 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: pgmap v1062: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 265 B/s, 2 objects/s recovering 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:25:18.889 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:18 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout:moved 3 / 396 (0.757576%) 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout:avg 66 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 7 -> 7.72442 (expected baseline 7.4162) 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.1 with 54 -> 54 pgs (0.818182 -> 0.818182 * mean) 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.7 with 76 -> 76 pgs (1.15152 -> 1.15152 * mean) 2024-03-04T16:25:19.018 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 0.05 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0091 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0100 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.6974 -> 0.6474 2024-03-04T16:25:19.019 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 weight 0.7821 -> 0.7321 2024-03-04T16:25:19.020 INFO:teuthology.orchestra.run.smithi038.stderr:no change 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: pgmap v1062: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 265 B/s, 2 objects/s recovering 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:25:19.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:25:19.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:18 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:25:19.210 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:25:19.017+0000 7fce816b4700 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 4.f deep-scrub starts 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 4.f deep-scrub ok 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: osdmap e482: 8 total, 8 up, 6 in 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 5.10 deep-scrub starts 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: from='client.36101 v1:172.21.15.38:0/2174355057' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 5.10 deep-scrub ok 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 4.f deep-scrub starts 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 4.f deep-scrub ok 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: osdmap e482: 8 total, 8 up, 6 in 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:25:19.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 5.10 deep-scrub starts 2024-03-04T16:25:19.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: from='client.36101 v1:172.21.15.38:0/2174355057' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:19.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 5.10 deep-scrub ok 2024-03-04T16:25:19.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:25:19.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:19 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:25:20.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 4.f deep-scrub starts 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 4.f deep-scrub ok 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: osdmap e482: 8 total, 8 up, 6 in 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 5.10 deep-scrub starts 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: from='client.36101 v1:172.21.15.38:0/2174355057' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:20.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 5.10 deep-scrub ok 2024-03-04T16:25:20.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:25:20.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:19 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:25:20.520 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-04T16:25:20.520 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight-by-utilization 110 0.05 2024-03-04T16:25:20.560 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:20.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:25:20.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:25:20.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[29779]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 502 B/s rd, 0 op/s; 53 B/s, 1 objects/s recovering 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[38113]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 502 B/s rd, 0 op/s; 53 B/s, 1 objects/s recovering 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:25:20.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:20 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:25:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:20 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:25:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:20 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:25:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:20 smithi176 ceph-mon[33619]: pgmap v1064: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 502 B/s rd, 0 op/s; 53 B/s, 1 objects/s recovering 2024-03-04T16:25:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:20 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:25:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:20 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:25:21.310 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:21.977 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:25:21.977 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:25:21.977 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:25:21.977 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:25:21.978 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: 3.b scrub starts 2024-03-04T16:25:21.978 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[29779]: 3.b scrub ok 2024-03-04T16:25:21.978 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:25:21.978 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:25:21.978 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:25:21.979 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:25:21.979 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: 3.b scrub starts 2024-03-04T16:25:21.979 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:21 smithi038 ceph-mon[38113]: 3.b scrub ok 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: 3.b scrub starts 2024-03-04T16:25:22.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:21 smithi176 ceph-mon[33619]: 3.b scrub ok 2024-03-04T16:25:22.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-04T16:25:22.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:25:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:25:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:25:22.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:22 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: pgmap v1065: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:25:23.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:22 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:25:23.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:23.627 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:23.627 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:23.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:25:23.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:25:23.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:25:23.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:25:23.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:25:23.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:25:23.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:25:23.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:23 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:25:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:23 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:25:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:23 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:25:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:23 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:25:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:23 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:25:24.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:24.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-04T16:25:24.811 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:24.933 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s; 12 B/s, 1 objects/s recovering 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:24.934 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s; 12 B/s, 1 objects/s recovering 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:24.935 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:25:24.936 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:24 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: pgmap v1066: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 898 B/s rd, 0 op/s; 12 B/s, 1 objects/s recovering 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:25.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]: dispatch 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:25:25.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:24 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout:moved 3 / 396 (0.757576%) 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout:avg 66 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout:stddev 6.68331 -> 7.50555 (expected baseline 7.4162) 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout:min osd.1 with 54 -> 54 pgs (0.818182 -> 0.818182 * mean) 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout:max osd.7 with 74 -> 74 pgs (1.12121 -> 1.12121 * mean) 2024-03-04T16:25:25.761 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:oload 110 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:max_change 0.05 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:max_change_osds 4 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:average_utilization 0.0091 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:overload_utilization 0.0100 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 weight 0.6974 -> 0.6474 2024-03-04T16:25:25.762 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 weight 0.7821 -> 0.7321 2024-03-04T16:25:25.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]': finished 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]': finished 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]': finished 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]': finished 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: osdmap e483: 8 total, 8 up, 6 in 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='client.16281 v1:172.21.15.38:0/2055192936' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: 3.9 scrub starts 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[29779]: 3.9 scrub ok 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]': finished 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]': finished 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]': finished 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]': finished 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: osdmap e483: 8 total, 8 up, 6 in 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='client.16281 v1:172.21.15.38:0/2055192936' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:25.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:25.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: 3.9 scrub starts 2024-03-04T16:25:25.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:25 smithi038 ceph-mon[38113]: 3.9 scrub ok 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.a", "id": [0, 6]}]': finished 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 2]}]': finished 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [7, 6]}]': finished 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.15", "id": [7, 1]}]': finished 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: osdmap e483: 8 total, 8 up, 6 in 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='client.16281 v1:172.21.15.38:0/2055192936' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]: dispatch 2024-03-04T16:25:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: 3.9 scrub starts 2024-03-04T16:25:26.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:25 smithi176 ceph-mon[33619]: 3.9 scrub ok 2024-03-04T16:25:27.030 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:27.031 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:27.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]': finished 2024-03-04T16:25:27.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[29779]: osdmap e484: 8 total, 8 up, 6 in 2024-03-04T16:25:27.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[29779]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:27.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[29779]: 5.f scrub starts 2024-03-04T16:25:27.031 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[29779]: 5.f scrub ok 2024-03-04T16:25:27.032 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]': finished 2024-03-04T16:25:27.032 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[38113]: osdmap e484: 8 total, 8 up, 6 in 2024-03-04T16:25:27.032 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[38113]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:27.032 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[38113]: 5.f scrub starts 2024-03-04T16:25:27.032 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:26 smithi038 ceph-mon[38113]: 5.f scrub ok 2024-03-04T16:25:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:26 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'42427','5':'47979'}"}]': finished 2024-03-04T16:25:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:26 smithi176 ceph-mon[33619]: osdmap e484: 8 total, 8 up, 6 in 2024-03-04T16:25:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:26 smithi176 ceph-mon[33619]: pgmap v1069: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 725 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:26 smithi176 ceph-mon[33619]: 5.f scrub starts 2024-03-04T16:25:27.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:26 smithi176 ceph-mon[33619]: 5.f scrub ok 2024-03-04T16:25:27.664 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-04T16:25:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: osdmap e485: 8 total, 8 up, 6 in 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:25:28.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: osdmap e485: 8 total, 8 up, 6 in 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:25:28.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:27 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: osdmap e485: 8 total, 8 up, 6 in 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:25:28.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:27 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: osdmap e486: 8 total, 8 up, 6 in 2024-03-04T16:25:29.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:25:29.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:25:29.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:25:29.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:28 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: osdmap e486: 8 total, 8 up, 6 in 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:25:29.119 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: pgmap v1071: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: osdmap e486: 8 total, 8 up, 6 in 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:25:29.120 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:28 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:25:29.783 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:29.783 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:30.426 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-04T16:25:30.671 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:31.104 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[29779]: 5.c scrub starts 2024-03-04T16:25:31.104 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[29779]: 5.c scrub ok 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[29779]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1018 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[38113]: 5.c scrub starts 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[38113]: 5.c scrub ok 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[38113]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1018 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:25:31.105 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:30 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:25:31.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:30 smithi176 ceph-mon[33619]: 5.c scrub starts 2024-03-04T16:25:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:30 smithi176 ceph-mon[33619]: 5.c scrub ok 2024-03-04T16:25:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:30 smithi176 ceph-mon[33619]: pgmap v1073: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1018 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:30 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:25:31.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:30 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:25:31.513 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-04T16:25:31.513 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-04T16:25:31.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd in 3 2024-03-04T16:25:31.854 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:25:32.067 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:31 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:25:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:31 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:25:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:31 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:25:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:31 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:25:32.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:31 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:25:32.274 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:32.274 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:32 smithi176 ceph-mon[33619]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 849 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:33.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:32 smithi176 ceph-mon[33619]: 6.f deep-scrub starts 2024-03-04T16:25:33.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:32 smithi038 ceph-mon[29779]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 849 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:33.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:32 smithi038 ceph-mon[29779]: 6.f deep-scrub starts 2024-03-04T16:25:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:32 smithi038 ceph-mon[38113]: pgmap v1074: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 849 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:33.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:32 smithi038 ceph-mon[38113]: 6.f deep-scrub starts 2024-03-04T16:25:33.695 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:33.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:25:33.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:25:33.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[29779]: 6.f deep-scrub ok 2024-03-04T16:25:33.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:25:33.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:25:33.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:33 smithi038 ceph-mon[38113]: 6.f deep-scrub ok 2024-03-04T16:25:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:33 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:25:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:33 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:25:34.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:33 smithi176 ceph-mon[33619]: 6.f deep-scrub ok 2024-03-04T16:25:34.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:34.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:25:34.496+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e487 definitely_dead 0 2024-03-04T16:25:34.841 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-04T16:25:34.880 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.3. 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: osdmap e487: 8 total, 8 up, 6 in 2024-03-04T16:25:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3219566909' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[29779]: from='client.36113 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:25:35.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:25:35.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:35.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:35.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: osdmap e487: 8 total, 8 up, 6 in 2024-03-04T16:25:35.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3219566909' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.247 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:34 smithi038 ceph-mon[38113]: from='client.36113 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: pgmap v1075: 132 pgs: 132 active+clean; 582 KiB data, 4.9 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: osdmap e487: 8 total, 8 up, 6 in 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3219566909' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:34 smithi176 ceph-mon[33619]: from='client.36113 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-04T16:25:35.606 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-04T16:25:35.606 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-04T16:25:35.606 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-04T16:25:35.606 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd in 4 2024-03-04T16:25:35.617 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:36.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:25:36.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:25:36.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[29779]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-04T16:25:36.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[29779]: osdmap e488: 8 total, 8 up, 7 in 2024-03-04T16:25:36.166 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[29779]: osdmap e489: 8 total, 8 up, 7 in 2024-03-04T16:25:36.167 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:25:36.167 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:25:36.167 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[38113]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-04T16:25:36.167 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[38113]: osdmap e488: 8 total, 8 up, 7 in 2024-03-04T16:25:36.167 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:35 smithi038 ceph-mon[38113]: osdmap e489: 8 total, 8 up, 7 in 2024-03-04T16:25:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:35 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:25:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:35 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:25:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:35 smithi176 ceph-mon[33619]: from='client.36113 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-04T16:25:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:35 smithi176 ceph-mon[33619]: osdmap e488: 8 total, 8 up, 7 in 2024-03-04T16:25:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:35 smithi176 ceph-mon[33619]: osdmap e489: 8 total, 8 up, 7 in 2024-03-04T16:25:37.011 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: 6.1 deep-scrub starts 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: pgmap v1078: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: 6.1 deep-scrub ok 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:25:37.214 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: 6.1 deep-scrub starts 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: pgmap v1078: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: 6.1 deep-scrub ok 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:25:37.215 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:36 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: 6.1 deep-scrub starts 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: pgmap v1078: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: 6.1 deep-scrub ok 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:25:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:36 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:25:37.620 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:37.620 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:25:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:25:38.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:25:38.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:37 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:25:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:37 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:25:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:37 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:25:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:37 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:25:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:37 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:25:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:37 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:25:39.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: pgmap v1080: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 5.0 scrub starts 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: 5.0 scrub ok 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:39.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: pgmap v1080: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 5.0 scrub starts 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: 5.0 scrub ok 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: pgmap v1080: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 5.0 scrub starts 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: 5.0 scrub ok 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:39.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-04T16:25:39.956 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:40.379 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:25:40.136+0000 7ff2a36cc700 -1 mon.a@0(leader).osd e489 definitely_dead 0 2024-03-04T16:25:40.935 INFO:teuthology.orchestra.run.smithi038.stderr:marked in osd.4. 2024-03-04T16:25:40.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: pgmap v1081: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 426 B/s, 9 objects/s recovering 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 5.1a deep-scrub starts 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 5.1a deep-scrub ok 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 5.1 scrub starts 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[29779]: 5.1 scrub ok 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: pgmap v1081: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 426 B/s, 9 objects/s recovering 2024-03-04T16:25:40.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 5.1a deep-scrub starts 2024-03-04T16:25:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 5.1a deep-scrub ok 2024-03-04T16:25:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-04T16:25:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 5.1 scrub starts 2024-03-04T16:25:40.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:40 smithi038 ceph-mon[38113]: 5.1 scrub ok 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: pgmap v1081: 132 pgs: 20 peering, 112 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 426 B/s, 9 objects/s recovering 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 5.1a deep-scrub starts 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 5.1a deep-scrub ok 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 5.1 scrub starts 2024-03-04T16:25:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:40 smithi176 ceph-mon[33619]: 5.1 scrub ok 2024-03-04T16:25:41.790 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-04T16:25:41.790 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[29779]: 5.19 deep-scrub starts 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[29779]: 5.19 deep-scrub ok 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[29779]: osdmap e490: 8 total, 8 up, 8 in 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[29779]: osdmap e491: 8 total, 8 up, 8 in 2024-03-04T16:25:42.039 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[38113]: 5.19 deep-scrub starts 2024-03-04T16:25:42.040 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[38113]: 5.19 deep-scrub ok 2024-03-04T16:25:42.040 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-04T16:25:42.040 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[38113]: osdmap e490: 8 total, 8 up, 8 in 2024-03-04T16:25:42.040 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:41 smithi038 ceph-mon[38113]: osdmap e491: 8 total, 8 up, 8 in 2024-03-04T16:25:42.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:41 smithi176 ceph-mon[33619]: 5.19 deep-scrub starts 2024-03-04T16:25:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:41 smithi176 ceph-mon[33619]: 5.19 deep-scrub ok 2024-03-04T16:25:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:41 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1662010268' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-04T16:25:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:41 smithi176 ceph-mon[33619]: osdmap e490: 8 total, 8 up, 8 in 2024-03-04T16:25:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:41 smithi176 ceph-mon[33619]: osdmap e491: 8 total, 8 up, 8 in 2024-03-04T16:25:42.388 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:42.391 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:42.397 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:43.168 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-04T16:25:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 6.11 scrub starts 2024-03-04T16:25:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 6.11 scrub ok 2024-03-04T16:25:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 591 B/s rd, 0 op/s; 449 B/s, 12 objects/s recovering 2024-03-04T16:25:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:25:43.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: Cluster is now healthy 2024-03-04T16:25:43.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 5.1f deep-scrub starts 2024-03-04T16:25:43.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 5.1f deep-scrub ok 2024-03-04T16:25:43.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:25:43.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 6.11 scrub starts 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 6.11 scrub ok 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 591 B/s rd, 0 op/s; 449 B/s, 12 objects/s recovering 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: Cluster is now healthy 2024-03-04T16:25:43.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 5.1f deep-scrub starts 2024-03-04T16:25:43.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 5.1f deep-scrub ok 2024-03-04T16:25:43.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:25:43.246 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:42 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 6.11 scrub starts 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 6.11 scrub ok 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: pgmap v1083: 132 pgs: 132 active+clean; 582 KiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 591 B/s rd, 0 op/s; 449 B/s, 12 objects/s recovering 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: Cluster is now healthy 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 5.1f deep-scrub starts 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 5.1f deep-scrub ok 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:25:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:42 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:25:43.570 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:43.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:44.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:25:44.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:43 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:25:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:43 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:25:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:43 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:25:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:43 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:25:44.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:43 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:25:44.400 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:44 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:45.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:44 smithi038 ceph-mon[29779]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 598 B/s, 20 objects/s recovering 2024-03-04T16:25:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:44 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:45.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:44 smithi038 ceph-mon[38113]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 598 B/s, 20 objects/s recovering 2024-03-04T16:25:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:44 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:44 smithi176 ceph-mon[33619]: pgmap v1085: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 645 B/s rd, 0 op/s; 598 B/s, 20 objects/s recovering 2024-03-04T16:25:45.546 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a: {} 2024-03-04T16:25:45.546 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b: {} 2024-03-04T16:25:45.550 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c: {} 2024-03-04T16:25:45.552 INFO:teuthology.orchestra.run.smithi038.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:25:45.552 INFO:teuthology.orchestra.run.smithi038.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:25:45.552 INFO:teuthology.orchestra.run.smithi038.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-04T16:25:46.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[29779]: 6.1a scrub starts 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[29779]: 6.1a scrub ok 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2961492429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[38113]: 6.1a scrub starts 2024-03-04T16:25:46.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[38113]: 6.1a scrub ok 2024-03-04T16:25:46.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:45 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2961492429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:25:46.300 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 0 1 2024-03-04T16:25:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:45 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:25:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:45 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:25:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:45 smithi176 ceph-mon[33619]: 6.1a scrub starts 2024-03-04T16:25:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:45 smithi176 ceph-mon[33619]: 6.1a scrub ok 2024-03-04T16:25:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:45 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2961492429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[29779]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 616 B/s, 20 objects/s recovering 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:25:47.042 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:46 smithi038 ceph-mon[38113]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 616 B/s, 20 objects/s recovering 2024-03-04T16:25:47.282 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:47.282 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:47.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:46 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:25:47.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:46 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:25:47.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:46 smithi176 ceph-mon[33619]: pgmap v1086: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 616 B/s, 20 objects/s recovering 2024-03-04T16:25:48.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-04T16:25:48.049 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:49.276 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[29779]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 270 B/s, 11 objects/s recovering 2024-03-04T16:25:49.276 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:25:49.276 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:25:49.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[38113]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 270 B/s, 11 objects/s recovering 2024-03-04T16:25:49.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:25:49.277 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:48 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:25:49.334 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:49.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:48 smithi176 ceph-mon[33619]: pgmap v1087: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 270 B/s, 11 objects/s recovering 2024-03-04T16:25:49.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:48 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:25:49.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:48 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:25:50.233 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:50 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:25:50.233 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:50 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:25:50.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:50 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:25:51.019 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.0 to 1 (10000) 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 693 B/s rd, 0 op/s; 182 B/s, 7 objects/s recovering 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: 6.16 scrub starts 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: 6.16 scrub ok 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:25:51.161 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 693 B/s rd, 0 op/s; 182 B/s, 7 objects/s recovering 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: 6.16 scrub starts 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: 6.16 scrub ok 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:25:51.162 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: pgmap v1088: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 693 B/s rd, 0 op/s; 182 B/s, 7 objects/s recovering 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: 6.16 scrub starts 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: 6.16 scrub ok 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:25:51.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:51.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-04T16:25:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:52.083 INFO:teuthology.orchestra.run.smithi038.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[29779]: 3.13 scrub starts 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[29779]: 3.13 scrub ok 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[29779]: osdmap e492: 8 total, 8 up, 8 in 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[38113]: 3.13 scrub starts 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[38113]: 3.13 scrub ok 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:52.084 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:52 smithi038 ceph-mon[38113]: osdmap e492: 8 total, 8 up, 8 in 2024-03-04T16:25:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:52 smithi176 ceph-mon[33619]: 3.13 scrub starts 2024-03-04T16:25:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:52 smithi176 ceph-mon[33619]: 3.13 scrub ok 2024-03-04T16:25:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:52 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2078281156' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:52.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:52 smithi176 ceph-mon[33619]: osdmap e492: 8 total, 8 up, 8 in 2024-03-04T16:25:52.953 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-04T16:25:53.007 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:53.198 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:25:53.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:53 smithi038 ceph-mon[29779]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 163 B/s, 6 objects/s recovering 2024-03-04T16:25:53.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:53 smithi038 ceph-mon[29779]: 6.b deep-scrub starts 2024-03-04T16:25:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:53 smithi038 ceph-mon[38113]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 163 B/s, 6 objects/s recovering 2024-03-04T16:25:53.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:53 smithi038 ceph-mon[38113]: 6.b deep-scrub starts 2024-03-04T16:25:53.321 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:53 smithi176 ceph-mon[33619]: pgmap v1090: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 623 B/s rd, 0 op/s; 163 B/s, 6 objects/s recovering 2024-03-04T16:25:53.321 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:53 smithi176 ceph-mon[33619]: 6.b deep-scrub starts 2024-03-04T16:25:53.743 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:25:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:25:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:25:53.975 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:25:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 6.b deep-scrub ok 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 5.9 deep-scrub starts 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 5.9 deep-scrub ok 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:54.144 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 6.b deep-scrub ok 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 5.9 deep-scrub starts 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 5.9 deep-scrub ok 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:54.145 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:25:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:25:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 6.b deep-scrub ok 2024-03-04T16:25:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 5.9 deep-scrub starts 2024-03-04T16:25:54.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 5.9 deep-scrub ok 2024-03-04T16:25:54.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:25:54.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:25:54.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:25:54.360 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:25:54.898 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:54.898 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:55.063 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: 3.1d scrub starts 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: 3.1d scrub ok 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3677700783' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:55 smithi176 ceph-mon[33619]: from='client.36143 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: 3.1d scrub starts 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: 3.1d scrub ok 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:25:55.426 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:25:55.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3677700783' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.427 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[29779]: from='client.36143 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.427 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:25:55.427 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: pgmap v1091: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 42 B/s, 1 objects/s recovering 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: 3.1d scrub starts 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: 3.1d scrub ok 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3677700783' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.428 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:55 smithi038 ceph-mon[38113]: from='client.36143 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-04T16:25:55.645 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-04T16:25:55.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 1 1 2024-03-04T16:25:55.900 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:25:55.996 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:56.271 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:56 smithi176 ceph-mon[33619]: from='client.36143 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:56.271 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:56 smithi176 ceph-mon[33619]: osdmap e493: 8 total, 8 up, 8 in 2024-03-04T16:25:56.271 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:56 smithi176 ceph-mon[33619]: osdmap e494: 8 total, 8 up, 8 in 2024-03-04T16:25:56.321 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[29779]: from='client.36143 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:56.321 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[29779]: osdmap e493: 8 total, 8 up, 8 in 2024-03-04T16:25:56.321 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[29779]: osdmap e494: 8 total, 8 up, 8 in 2024-03-04T16:25:56.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[38113]: from='client.36143 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-04T16:25:56.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[38113]: osdmap e493: 8 total, 8 up, 8 in 2024-03-04T16:25:56.322 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:56 smithi038 ceph-mon[38113]: osdmap e494: 8 total, 8 up, 8 in 2024-03-04T16:25:57.539 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:25:57.540 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 6.e deep-scrub starts 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 6.e deep-scrub ok 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:25:57.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:57 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:25:57.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:25:57.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:25:57.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:25:57.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 6.e deep-scrub starts 2024-03-04T16:25:57.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 6.e deep-scrub ok 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 6.e deep-scrub starts 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 6.e deep-scrub ok 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:25:57.681 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:57 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:25:58.278 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-04T16:25:58.499 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.1 to 1 (10000) 2024-03-04T16:25:58.526 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:25:58.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:58 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:25:58.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:58 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:25:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:58 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:25:59.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-04T16:25:59.356 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:25:59.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[38113]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-04T16:25:59.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:25:59.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[38113]: osdmap e495: 8 total, 8 up, 8 in 2024-03-04T16:25:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[29779]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-04T16:25:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:25:59.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:25:59 smithi038 ceph-mon[29779]: osdmap e495: 8 total, 8 up, 8 in 2024-03-04T16:25:59.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:59 smithi176 ceph-mon[33619]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 755 B/s rd, 0 op/s 2024-03-04T16:25:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:59 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/694478018' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:25:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:25:59 smithi176 ceph-mon[33619]: osdmap e495: 8 total, 8 up, 8 in 2024-03-04T16:26:00.293 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:00.293 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: osdmap e496: 8 total, 8 up, 8 in 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 3.11 deep-scrub starts 2024-03-04T16:26:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:00 smithi176 ceph-mon[33619]: 3.11 deep-scrub ok 2024-03-04T16:26:00.874 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: osdmap e496: 8 total, 8 up, 8 in 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 3.11 deep-scrub starts 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[29779]: 3.11 deep-scrub ok 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:26:00.875 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: osdmap e496: 8 total, 8 up, 8 in 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: pgmap v1098: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 3.11 deep-scrub starts 2024-03-04T16:26:00.876 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:00 smithi038 ceph-mon[38113]: 3.11 deep-scrub ok 2024-03-04T16:26:00.911 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-04T16:26:01.152 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:01.552 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-04T16:26:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:01 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:26:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:01 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:26:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:01 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1752982362' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:01 smithi176 ceph-mon[33619]: from='client.36155 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:01.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:26:01.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:26:01.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1752982362' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:01.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[29779]: from='client.36155 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:01.910 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:26:01.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:26:01.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1752982362' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:01.911 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:01 smithi038 ceph-mon[38113]: from='client.36155 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-04T16:26:02.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 2 1 2024-03-04T16:26:02.478 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:02.689 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:02 smithi176 ceph-mon[33619]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:26:02.690 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:02 smithi176 ceph-mon[33619]: osdmap e497: 8 total, 8 up, 8 in 2024-03-04T16:26:02.690 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:02 smithi176 ceph-mon[33619]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-04T16:26:02.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[29779]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:26:02.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[29779]: osdmap e497: 8 total, 8 up, 8 in 2024-03-04T16:26:02.880 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[29779]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-04T16:26:02.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[38113]: from='client.36155 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-04T16:26:02.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[38113]: osdmap e497: 8 total, 8 up, 8 in 2024-03-04T16:26:02.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:02 smithi038 ceph-mon[38113]: pgmap v1100: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-04T16:26:02.918 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:02.918 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:03.568 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:03.591 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-04T16:26:03.837 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:26:03.840 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:26:03.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:26:03.841 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:03 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:26:03.857 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:03 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:26:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:03 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:26:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:03 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:26:03.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:03 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:26:04.587 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.2 to 1 (10000) 2024-03-04T16:26:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:04 smithi176 ceph-mon[33619]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-04T16:26:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:04 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3486180086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:04.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:04 smithi176 ceph-mon[33619]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:04.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[29779]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3486180086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[29779]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[38113]: pgmap v1101: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3486180086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:04.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:04 smithi038 ceph-mon[38113]: from='client.36161 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:05.476 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-04T16:26:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:05.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: osdmap e498: 8 total, 8 up, 8 in 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: 4.b scrub starts 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: 4.b scrub ok 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: osdmap e498: 8 total, 8 up, 8 in 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: 4.b scrub starts 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: 4.b scrub ok 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:26:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:05 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: from='client.36161 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: osdmap e498: 8 total, 8 up, 8 in 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: 4.b scrub starts 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: 4.b scrub ok 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:26:05.745 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:05 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:26:05.795 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:05.795 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:06.511 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-04T16:26:06.759 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:06.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:06 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:06 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:06 smithi176 ceph-mon[33619]: osdmap e499: 8 total, 8 up, 8 in 2024-03-04T16:26:06.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:06 smithi176 ceph-mon[33619]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:06.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:06.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:06.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[38113]: osdmap e499: 8 total, 8 up, 8 in 2024-03-04T16:26:06.943 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[38113]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:06.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:06.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:06.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[29779]: osdmap e499: 8 total, 8 up, 8 in 2024-03-04T16:26:06.944 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:06 smithi038 ceph-mon[29779]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:07 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:07 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:07 smithi176 ceph-mon[33619]: osdmap e500: 8 total, 8 up, 8 in 2024-03-04T16:26:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:07 smithi176 ceph-mon[33619]: 3.4 scrub starts 2024-03-04T16:26:07.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:07 smithi176 ceph-mon[33619]: 3.4 scrub ok 2024-03-04T16:26:07.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:07.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[38113]: osdmap e500: 8 total, 8 up, 8 in 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[38113]: 3.4 scrub starts 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[38113]: 3.4 scrub ok 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[29779]: osdmap e500: 8 total, 8 up, 8 in 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[29779]: 3.4 scrub starts 2024-03-04T16:26:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:07 smithi038 ceph-mon[29779]: 3.4 scrub ok 2024-03-04T16:26:08.583 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:08.583 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:08.613 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-04T16:26:08.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:08 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:26:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:08 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:26:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:08 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:08.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:08 smithi176 ceph-mon[33619]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:26:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:26:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:08.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[29779]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:08.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:26:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:26:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-04T16:26:08.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:08 smithi038 ceph-mon[38113]: pgmap v1106: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 101 B/s, 1 objects/s recovering 2024-03-04T16:26:09.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-04T16:26:09.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 3 1 2024-03-04T16:26:09.557 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:09.572 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:09.902 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:09 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:09.902 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:09 smithi176 ceph-mon[33619]: osdmap e501: 8 total, 8 up, 8 in 2024-03-04T16:26:09.902 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:09 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:09.902 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:09 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:26:09.902 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:09 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[29779]: osdmap e501: 8 total, 8 up, 8 in 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2862361598' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[38113]: osdmap e501: 8 total, 8 up, 8 in 2024-03-04T16:26:09.913 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:09.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:26:09.914 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:09 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:26:11.207 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:11.208 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:11.628 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.3 to 1 (10000) 2024-03-04T16:26:11.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: 4.1b deep-scrub starts 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: 4.1b deep-scrub ok 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/4216196412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[29779]: from='client.36173 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: 4.1b deep-scrub starts 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: 4.1b deep-scrub ok 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:11.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:11.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/4216196412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:11 smithi038 ceph-mon[38113]: from='client.36173 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: pgmap v1108: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 982 B/s rd, 0 op/s 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: 4.1b deep-scrub starts 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: 4.1b deep-scrub ok 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/4216196412' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.830 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:11 smithi176 ceph-mon[33619]: from='client.36173 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:11.857 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-04T16:26:12.098 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:12.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-04T16:26:12.613 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: osdmap e502: 8 total, 8 up, 8 in 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:26:12.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:12 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:26:12.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:26:12.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:26:12.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: osdmap e502: 8 total, 8 up, 8 in 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: from='client.36173 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: osdmap e502: 8 total, 8 up, 8 in 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: pgmap v1110: 132 pgs: 132 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:26:12.921 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:12 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:26:13.641 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:13.875 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:13.875 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:13.884 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:13 smithi176 ceph-mon[33619]: 6.10 deep-scrub starts 2024-03-04T16:26:13.885 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:13 smithi176 ceph-mon[33619]: 6.10 deep-scrub ok 2024-03-04T16:26:13.885 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:13.927 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:13 smithi038 ceph-mon[29779]: 6.10 deep-scrub starts 2024-03-04T16:26:13.927 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:13 smithi038 ceph-mon[29779]: 6.10 deep-scrub ok 2024-03-04T16:26:13.928 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:13 smithi038 ceph-mon[38113]: 6.10 deep-scrub starts 2024-03-04T16:26:13.928 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:13 smithi038 ceph-mon[38113]: 6.10 deep-scrub ok 2024-03-04T16:26:14.565 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: osdmap e503: 8 total, 8 up, 8 in 2024-03-04T16:26:14.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: pgmap v1112: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: osdmap e503: 8 total, 8 up, 8 in 2024-03-04T16:26:14.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: pgmap v1112: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:14.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:14.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:14 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:14.811 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: osdmap e503: 8 total, 8 up, 8 in 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: pgmap v1112: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:14.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:14 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:15.704 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-04T16:26:15.941 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 5.17 scrub starts 2024-03-04T16:26:15.941 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 5.17 scrub ok 2024-03-04T16:26:15.941 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: osdmap e504: 8 total, 8 up, 8 in 2024-03-04T16:26:15.941 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2744364571' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.941 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: from='client.36179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.942 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:26:15.942 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:26:15.942 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:26:15.942 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:15 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:26:15.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 5.17 scrub starts 2024-03-04T16:26:15.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 5.17 scrub ok 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: osdmap e504: 8 total, 8 up, 8 in 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2744364571' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: from='client.36179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 5.17 scrub starts 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 5.17 scrub ok 2024-03-04T16:26:15.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: osdmap e504: 8 total, 8 up, 8 in 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2744364571' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: from='client.36179 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:26:15.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:15 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:26:16.461 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 4 1 2024-03-04T16:26:16.681 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:16.681 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:16.717 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:16.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[29779]: from='client.36179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:16.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[29779]: osdmap e505: 8 total, 8 up, 8 in 2024-03-04T16:26:16.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[29779]: 3.12 deep-scrub starts 2024-03-04T16:26:16.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[29779]: 3.12 deep-scrub ok 2024-03-04T16:26:16.724 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[29779]: pgmap v1115: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:26:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:16 smithi176 ceph-mon[33619]: from='client.36179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:16 smithi176 ceph-mon[33619]: osdmap e505: 8 total, 8 up, 8 in 2024-03-04T16:26:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:16 smithi176 ceph-mon[33619]: 3.12 deep-scrub starts 2024-03-04T16:26:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:16 smithi176 ceph-mon[33619]: 3.12 deep-scrub ok 2024-03-04T16:26:16.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:16 smithi176 ceph-mon[33619]: pgmap v1115: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:26:16.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[38113]: from='client.36179 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-04T16:26:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[38113]: osdmap e505: 8 total, 8 up, 8 in 2024-03-04T16:26:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[38113]: 3.12 deep-scrub starts 2024-03-04T16:26:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[38113]: 3.12 deep-scrub ok 2024-03-04T16:26:16.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:16 smithi038 ceph-mon[38113]: pgmap v1115: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:26:17.431 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-04T16:26:17.670 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[29779]: 3.6 deep-scrub starts 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[29779]: 3.6 deep-scrub ok 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[29779]: osdmap e506: 8 total, 8 up, 8 in 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[38113]: 3.6 deep-scrub starts 2024-03-04T16:26:17.930 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[38113]: 3.6 deep-scrub ok 2024-03-04T16:26:17.931 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[38113]: osdmap e506: 8 total, 8 up, 8 in 2024-03-04T16:26:17.931 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:26:17.931 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:17 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:26:18.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:17 smithi176 ceph-mon[33619]: 3.6 deep-scrub starts 2024-03-04T16:26:18.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:17 smithi176 ceph-mon[33619]: 3.6 deep-scrub ok 2024-03-04T16:26:18.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:17 smithi176 ceph-mon[33619]: osdmap e506: 8 total, 8 up, 8 in 2024-03-04T16:26:18.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:17 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:26:18.058 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:17 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:26:18.725 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.4 to 1 (10000) 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 5.1e deep-scrub starts 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 5.1e deep-scrub ok 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: pgmap v1117: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:26:18.837 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:26:18.838 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:26:18.838 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:18 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 5.1e deep-scrub starts 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 5.1e deep-scrub ok 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: pgmap v1117: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:26:18.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 5.1e deep-scrub starts 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 5.1e deep-scrub ok 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: pgmap v1117: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:26:18.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:26:18.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:26:18.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:26:18.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:26:18.995 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:18 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:19.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-04T16:26:19.408 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:19.408 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:19.664 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[38113]: osdmap e507: 8 total, 8 up, 8 in 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[38113]: 6.2 scrub starts 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[38113]: 6.2 scrub ok 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[29779]: osdmap e507: 8 total, 8 up, 8 in 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[29779]: 6.2 scrub starts 2024-03-04T16:26:19.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:19 smithi038 ceph-mon[29779]: 6.2 scrub ok 2024-03-04T16:26:20.003 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:19 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1525922719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:20.003 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:19 smithi176 ceph-mon[33619]: osdmap e507: 8 total, 8 up, 8 in 2024-03-04T16:26:20.003 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:19 smithi176 ceph-mon[33619]: 6.2 scrub starts 2024-03-04T16:26:20.003 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:19 smithi176 ceph-mon[33619]: 6.2 scrub ok 2024-03-04T16:26:20.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-04T16:26:20.283 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:20.915 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:26:20.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:26:20.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 5.3 scrub starts 2024-03-04T16:26:20.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 5.3 scrub ok 2024-03-04T16:26:20.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-04T16:26:20.916 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 3.14 scrub starts 2024-03-04T16:26:20.917 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 3.14 scrub ok 2024-03-04T16:26:20.917 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:26:20.917 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:26:20.917 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:26:20.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:26:20.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 5.3 scrub starts 2024-03-04T16:26:20.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 5.3 scrub ok 2024-03-04T16:26:20.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-04T16:26:20.918 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 3.14 scrub starts 2024-03-04T16:26:20.919 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 3.14 scrub ok 2024-03-04T16:26:20.919 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:26:20.919 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:20 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 5.3 scrub starts 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 5.3 scrub ok 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 3.14 scrub starts 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 3.14 scrub ok 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:26:21.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:20 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:26:21.785 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-04T16:26:21.880 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: 6.c deep-scrub starts 2024-03-04T16:26:21.880 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: 6.c deep-scrub ok 2024-03-04T16:26:21.880 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:21.881 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:21.881 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1743681950' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:21.881 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:21 smithi176 ceph-mon[33619]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:22.006 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:22.006 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:22.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: 6.c deep-scrub starts 2024-03-04T16:26:22.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: 6.c deep-scrub ok 2024-03-04T16:26:22.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:22.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:22.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1743681950' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[29779]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: 6.c deep-scrub starts 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: 6.c deep-scrub ok 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1743681950' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:22.140 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:21 smithi038 ceph-mon[38113]: from='client.36191 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-04T16:26:22.427 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 5 1 2024-03-04T16:26:22.580 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-04T16:26:22.676 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:22.838 INFO:teuthology.orchestra.run.smithi176.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.b/config 2024-03-04T16:26:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:22 smithi176 ceph-mon[33619]: 4.13 deep-scrub starts 2024-03-04T16:26:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:22 smithi176 ceph-mon[33619]: 4.13 deep-scrub ok 2024-03-04T16:26:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:22 smithi176 ceph-mon[33619]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:22 smithi176 ceph-mon[33619]: osdmap e508: 8 total, 8 up, 8 in 2024-03-04T16:26:22.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:22 smithi176 ceph-mon[33619]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[29779]: 4.13 deep-scrub starts 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[29779]: 4.13 deep-scrub ok 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[29779]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[29779]: osdmap e508: 8 total, 8 up, 8 in 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[29779]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[38113]: 4.13 deep-scrub starts 2024-03-04T16:26:23.077 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[38113]: 4.13 deep-scrub ok 2024-03-04T16:26:23.078 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[38113]: from='client.36191 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-04T16:26:23.078 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[38113]: osdmap e508: 8 total, 8 up, 8 in 2024-03-04T16:26:23.078 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:22 smithi038 ceph-mon[38113]: pgmap v1121: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-04T16:26:23.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:23.943 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:23 smithi176 ceph-mon[33619]: 3.1a deep-scrub starts 2024-03-04T16:26:23.944 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:23 smithi176 ceph-mon[33619]: 3.1a deep-scrub ok 2024-03-04T16:26:23.944 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:24.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:23 smithi038 ceph-mon[29779]: 3.1a deep-scrub starts 2024-03-04T16:26:24.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:23 smithi038 ceph-mon[29779]: 3.1a deep-scrub ok 2024-03-04T16:26:24.103 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:23 smithi038 ceph-mon[38113]: 3.1a deep-scrub starts 2024-03-04T16:26:24.104 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:23 smithi038 ceph-mon[38113]: 3.1a deep-scrub ok 2024-03-04T16:26:24.505 INFO:teuthology.orchestra.run.smithi176.stdout:ERROR: (22) Invalid argument 2024-03-04T16:26:24.505 INFO:teuthology.orchestra.run.smithi176.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-04T16:26:24.862 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.5 to 1 (10000) 2024-03-04T16:26:25.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:25.110 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: 3.17 scrub starts 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: 3.17 scrub ok 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1032975626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='client.36197 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.111 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:24 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: 3.17 scrub starts 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: 3.17 scrub ok 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1032975626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='client.36197 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: 3.17 scrub starts 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: 3.17 scrub ok 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1032975626' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='client.36197 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:24 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-04T16:26:25.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-04T16:26:25.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: 4.1e scrub starts 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: 4.1e scrub ok 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: 5.7 scrub starts 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: from='client.36197 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[38113]: osdmap e509: 8 total, 8 up, 8 in 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: 4.1e scrub starts 2024-03-04T16:26:25.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: 4.1e scrub ok 2024-03-04T16:26:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: 5.7 scrub starts 2024-03-04T16:26:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: from='client.36197 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:26:25.994 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:25 smithi038 ceph-mon[29779]: osdmap e509: 8 total, 8 up, 8 in 2024-03-04T16:26:26.018 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: 4.1e scrub starts 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: 4.1e scrub ok 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: 5.7 scrub starts 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: from='client.36197 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-04T16:26:26.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:25 smithi176 ceph-mon[33619]: osdmap e509: 8 total, 8 up, 8 in 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 4.c scrub starts 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 4.c scrub ok 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: pgmap v1124: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.10 scrub starts 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.10 scrub ok 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: osdmap e510: 8 total, 8 up, 8 in 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.7 deep-scrub starts 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.7 deep-scrub ok 2024-03-04T16:26:27.162 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.b scrub starts 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[38113]: 5.b scrub ok 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 4.c scrub starts 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 4.c scrub ok 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: pgmap v1124: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.10 scrub starts 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.10 scrub ok 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: osdmap e510: 8 total, 8 up, 8 in 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.7 deep-scrub starts 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.7 deep-scrub ok 2024-03-04T16:26:27.163 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.b scrub starts 2024-03-04T16:26:27.164 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:26 smithi038 ceph-mon[29779]: 5.b scrub ok 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 4.c scrub starts 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 4.c scrub ok 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: pgmap v1124: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 579 B/s rd, 0 op/s 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.10 scrub starts 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.10 scrub ok 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: osdmap e510: 8 total, 8 up, 8 in 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.7 deep-scrub starts 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.7 deep-scrub ok 2024-03-04T16:26:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.b scrub starts 2024-03-04T16:26:27.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:26 smithi176 ceph-mon[33619]: 5.b scrub ok 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 3.1e scrub starts 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 3.1e scrub ok 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 3.d deep-scrub starts 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 3.d deep-scrub ok 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 6.14 scrub starts 2024-03-04T16:26:28.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[29779]: 6.14 scrub ok 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 3.1e scrub starts 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 3.1e scrub ok 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 3.d deep-scrub starts 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 3.d deep-scrub ok 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 6.14 scrub starts 2024-03-04T16:26:28.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:27 smithi038 ceph-mon[38113]: 6.14 scrub ok 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 3.1e scrub starts 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 3.1e scrub ok 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 3.d deep-scrub starts 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 3.d deep-scrub ok 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 6.14 scrub starts 2024-03-04T16:26:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:27 smithi176 ceph-mon[33619]: 6.14 scrub ok 2024-03-04T16:26:28.919 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-04T16:26:29.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:28 smithi038 ceph-mon[29779]: pgmap v1126: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-04T16:26:29.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:28 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:29.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:28 smithi038 ceph-mon[38113]: pgmap v1126: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-04T16:26:29.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:28 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:28 smithi176 ceph-mon[33619]: pgmap v1126: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-04T16:26:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:28 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-04T16:26:29.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 6 1 2024-03-04T16:26:29.943 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:30.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: 6.13 scrub starts 2024-03-04T16:26:30.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: 6.13 scrub ok 2024-03-04T16:26:30.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:30.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: osdmap e511: 8 total, 8 up, 8 in 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: 4.e deep-scrub starts 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[29779]: 4.e deep-scrub ok 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: 6.13 scrub starts 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: 6.13 scrub ok 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: osdmap e511: 8 total, 8 up, 8 in 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: 4.e deep-scrub starts 2024-03-04T16:26:30.200 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:29 smithi038 ceph-mon[38113]: 4.e deep-scrub ok 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: 6.13 scrub starts 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: 6.13 scrub ok 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1012385866' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: osdmap e511: 8 total, 8 up, 8 in 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: 4.e deep-scrub starts 2024-03-04T16:26:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:29 smithi176 ceph-mon[33619]: 4.e deep-scrub ok 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[29779]: 6.18 scrub starts 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[29779]: pgmap v1128: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[29779]: 6.18 scrub ok 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[38113]: 6.18 scrub starts 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[38113]: pgmap v1128: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-03-04T16:26:31.203 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:30 smithi038 ceph-mon[38113]: 6.18 scrub ok 2024-03-04T16:26:31.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:30 smithi176 ceph-mon[33619]: 6.18 scrub starts 2024-03-04T16:26:31.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:30 smithi176 ceph-mon[33619]: pgmap v1128: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 1 objects/s recovering 2024-03-04T16:26:31.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:30 smithi176 ceph-mon[33619]: 6.18 scrub ok 2024-03-04T16:26:31.943 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.6 to 1 (10000) 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 4.19 scrub starts 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 4.19 scrub ok 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 4.7 deep-scrub starts 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 4.7 deep-scrub ok 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 6.a scrub starts 2024-03-04T16:26:32.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 6.a scrub ok 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 5.1c scrub starts 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: 5.1c scrub ok 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/527896441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[29779]: from='client.36209 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 4.19 scrub starts 2024-03-04T16:26:32.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 4.19 scrub ok 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 4.7 deep-scrub starts 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 4.7 deep-scrub ok 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 6.a scrub starts 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 6.a scrub ok 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 5.1c scrub starts 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: 5.1c scrub ok 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/527896441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:31 smithi038 ceph-mon[38113]: from='client.36209 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 4.19 scrub starts 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 4.19 scrub ok 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 4.7 deep-scrub starts 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 4.7 deep-scrub ok 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 6.a scrub starts 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 6.a scrub ok 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 5.1c scrub starts 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: 5.1c scrub ok 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/527896441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:31 smithi176 ceph-mon[33619]: from='client.36209 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:32.727 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-04T16:26:32.976 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:32.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: 3.1 scrub starts 2024-03-04T16:26:32.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: 3.1 scrub ok 2024-03-04T16:26:32.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: 1.0 scrub starts 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: 1.0 scrub ok 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: from='client.36209 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[38113]: osdmap e512: 8 total, 8 up, 8 in 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: 3.1 scrub starts 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: 3.1 scrub ok 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: 1.0 scrub starts 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: 1.0 scrub ok 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: from='client.36209 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:32.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:32 smithi038 ceph-mon[29779]: osdmap e512: 8 total, 8 up, 8 in 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: 3.1 scrub starts 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: 3.1 scrub ok 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: pgmap v1129: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 736 B/s rd, 0 op/s; 31 B/s, 1 objects/s recovering 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: 1.0 scrub starts 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: 1.0 scrub ok 2024-03-04T16:26:33.259 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: from='client.36209 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:33.260 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:32 smithi176 ceph-mon[33619]: osdmap e512: 8 total, 8 up, 8 in 2024-03-04T16:26:33.379 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:33.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:34.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 4.d scrub starts 2024-03-04T16:26:34.184 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 4.d scrub ok 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 4.3 scrub starts 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 6.4 scrub starts 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 6.4 scrub ok 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[29779]: 4.3 scrub ok 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 4.d scrub starts 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 4.d scrub ok 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 4.3 scrub starts 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 6.4 scrub starts 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 6.4 scrub ok 2024-03-04T16:26:34.185 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:33 smithi038 ceph-mon[38113]: 4.3 scrub ok 2024-03-04T16:26:34.271 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 4.d scrub starts 2024-03-04T16:26:34.272 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 4.d scrub ok 2024-03-04T16:26:34.272 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 4.3 scrub starts 2024-03-04T16:26:34.272 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 6.4 scrub starts 2024-03-04T16:26:34.272 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 6.4 scrub ok 2024-03-04T16:26:34.272 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:33 smithi176 ceph-mon[33619]: 4.3 scrub ok 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[29779]: 4.2 scrub starts 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[29779]: 4.2 scrub ok 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[29779]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[38113]: 4.2 scrub starts 2024-03-04T16:26:35.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[38113]: 4.2 scrub ok 2024-03-04T16:26:35.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:34 smithi038 ceph-mon[38113]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:34 smithi176 ceph-mon[33619]: 4.2 scrub starts 2024-03-04T16:26:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:34 smithi176 ceph-mon[33619]: 4.2 scrub ok 2024-03-04T16:26:35.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:34 smithi176 ceph-mon[33619]: pgmap v1131: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[38113]: 3.0 deep-scrub starts 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[38113]: 3.0 deep-scrub ok 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[29779]: 3.0 deep-scrub starts 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[29779]: 3.0 deep-scrub ok 2024-03-04T16:26:36.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:35 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:35 smithi176 ceph-mon[33619]: 3.0 deep-scrub starts 2024-03-04T16:26:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:35 smithi176 ceph-mon[33619]: 3.0 deep-scrub ok 2024-03-04T16:26:36.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:35 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-04T16:26:36.743 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd reweight 7 1 2024-03-04T16:26:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:37.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[29779]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:37.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[29779]: osdmap e513: 8 total, 8 up, 8 in 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[29779]: 3.15 scrub starts 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[29779]: 3.15 scrub ok 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[38113]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[38113]: osdmap e513: 8 total, 8 up, 8 in 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[38113]: 3.15 scrub starts 2024-03-04T16:26:37.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:36 smithi038 ceph-mon[38113]: 3.15 scrub ok 2024-03-04T16:26:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:36 smithi176 ceph-mon[33619]: pgmap v1132: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-04T16:26:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:36 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1789135648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-04T16:26:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:36 smithi176 ceph-mon[33619]: osdmap e513: 8 total, 8 up, 8 in 2024-03-04T16:26:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:36 smithi176 ceph-mon[33619]: 3.15 scrub starts 2024-03-04T16:26:37.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:36 smithi176 ceph-mon[33619]: 3.15 scrub ok 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[29779]: 5.14 scrub starts 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[29779]: 5.14 scrub ok 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[29779]: 5.d scrub starts 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[29779]: 5.d scrub ok 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[38113]: 5.14 scrub starts 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[38113]: 5.14 scrub ok 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[38113]: 5.d scrub starts 2024-03-04T16:26:38.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:37 smithi038 ceph-mon[38113]: 5.d scrub ok 2024-03-04T16:26:38.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:37 smithi176 ceph-mon[33619]: 5.14 scrub starts 2024-03-04T16:26:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:37 smithi176 ceph-mon[33619]: 5.14 scrub ok 2024-03-04T16:26:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:37 smithi176 ceph-mon[33619]: 5.d scrub starts 2024-03-04T16:26:38.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:37 smithi176 ceph-mon[33619]: 5.d scrub ok 2024-03-04T16:26:38.984 INFO:teuthology.orchestra.run.smithi038.stderr:reweighted osd.7 to 1 (10000) 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: 6.d deep-scrub starts 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: osdmap e514: 8 total, 8 up, 8 in 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:39.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:26:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: 6.d deep-scrub starts 2024-03-04T16:26:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: osdmap e514: 8 total, 8 up, 8 in 2024-03-04T16:26:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:39.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:38 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: pgmap v1134: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: 6.d deep-scrub starts 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: osdmap e514: 8 total, 8 up, 8 in 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:39.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:38 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:39.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-04T16:26:39.935 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:40.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[29779]: 6.d deep-scrub ok 2024-03-04T16:26:40.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[29779]: 5.13 scrub starts 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[29779]: 5.13 scrub ok 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[29779]: osdmap e515: 8 total, 8 up, 8 in 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[38113]: 6.d deep-scrub ok 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[38113]: 5.13 scrub starts 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[38113]: 5.13 scrub ok 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:40.323 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:40 smithi038 ceph-mon[38113]: osdmap e515: 8 total, 8 up, 8 in 2024-03-04T16:26:40.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:40 smithi176 ceph-mon[33619]: 6.d deep-scrub ok 2024-03-04T16:26:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:40 smithi176 ceph-mon[33619]: 5.13 scrub starts 2024-03-04T16:26:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:40 smithi176 ceph-mon[33619]: 5.13 scrub ok 2024-03-04T16:26:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:40 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1265335985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:40.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:40 smithi176 ceph-mon[33619]: osdmap e515: 8 total, 8 up, 8 in 2024-03-04T16:26:41.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 6.6 scrub starts 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 6.6 scrub ok 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 6.12 scrub starts 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 6.12 scrub ok 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 5.6 scrub starts 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[29779]: 5.6 scrub ok 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 6.6 scrub starts 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 6.6 scrub ok 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:41.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 6.12 scrub starts 2024-03-04T16:26:41.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 6.12 scrub ok 2024-03-04T16:26:41.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 5.6 scrub starts 2024-03-04T16:26:41.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:41 smithi038 ceph-mon[38113]: 5.6 scrub ok 2024-03-04T16:26:41.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 6.6 scrub starts 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 6.6 scrub ok 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 6.12 scrub starts 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 6.12 scrub ok 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 5.6 scrub starts 2024-03-04T16:26:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:41 smithi176 ceph-mon[33619]: 5.6 scrub ok 2024-03-04T16:26:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:42 smithi176 ceph-mon[33619]: 3.18 scrub starts 2024-03-04T16:26:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:42 smithi176 ceph-mon[33619]: 3.18 scrub ok 2024-03-04T16:26:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:42 smithi176 ceph-mon[33619]: 2.1 scrub starts 2024-03-04T16:26:42.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:42 smithi176 ceph-mon[33619]: 2.1 scrub ok 2024-03-04T16:26:42.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[29779]: 3.18 scrub starts 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[29779]: 3.18 scrub ok 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[29779]: 2.1 scrub starts 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[29779]: 2.1 scrub ok 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[38113]: 3.18 scrub starts 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[38113]: 3.18 scrub ok 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[38113]: 2.1 scrub starts 2024-03-04T16:26:42.493 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:42 smithi038 ceph-mon[38113]: 2.1 scrub ok 2024-03-04T16:26:43.044 INFO:teuthology.orchestra.run.smithi038.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[38113]: 3.b deep-scrub starts 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[38113]: 3.b deep-scrub ok 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[38113]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/12182318' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[38113]: from='client.36227 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[29779]: 3.b deep-scrub starts 2024-03-04T16:26:43.306 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[29779]: 3.b deep-scrub ok 2024-03-04T16:26:43.307 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[29779]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:43.307 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/12182318' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.307 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-mon[29779]: from='client.36227 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-mon[33619]: 3.b deep-scrub starts 2024-03-04T16:26:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-mon[33619]: 3.b deep-scrub ok 2024-03-04T16:26:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-mon[33619]: pgmap v1138: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:26:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/12182318' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-mon[33619]: from='client.36227 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-04T16:26:43.719 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:43 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:43] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:43.751 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-04T16:26:43.751 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-04T16:26:43.751 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-04T16:26:43.751 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-04T16:26:43.751 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-04T16:26:43.752 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:26:44.007 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:44.054 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:44.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: 6.9 scrub starts 2024-03-04T16:26:44.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: 6.1c scrub starts 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: 6.1c scrub ok 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: 6.9 scrub ok 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: from='client.36227 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[29779]: osdmap e516: 8 total, 8 up, 8 in 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: 6.9 scrub starts 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: 6.1c scrub starts 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: 6.1c scrub ok 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: 6.9 scrub ok 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: from='client.36227 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:44.309 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:44 smithi038 ceph-mon[38113]: osdmap e516: 8 total, 8 up, 8 in 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: 6.9 scrub starts 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: 6.1c scrub starts 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: 6.1c scrub ok 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: 6.9 scrub ok 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: from='client.36227 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-04T16:26:44.316 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:44 smithi176 ceph-mon[33619]: osdmap e516: 8 total, 8 up, 8 in 2024-03-04T16:26:45.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:45.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: 3.3 scrub starts 2024-03-04T16:26:45.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: 3.3 scrub ok 2024-03-04T16:26:45.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:45.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: 4.4 scrub starts 2024-03-04T16:26:45.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[29779]: 4.4 scrub ok 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: 3.3 scrub starts 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: 3.3 scrub ok 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: 4.4 scrub starts 2024-03-04T16:26:45.245 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:45 smithi038 ceph-mon[38113]: 4.4 scrub ok 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: 3.3 scrub starts 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: 3.3 scrub ok 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: pgmap v1140: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: 4.4 scrub starts 2024-03-04T16:26:45.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:45 smithi176 ceph-mon[33619]: 4.4 scrub ok 2024-03-04T16:26:46.037 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:26:46.037 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":516,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:57:15.255572+0000","modified":"2024-03-04T16:26:43.023067+0000","last_up_change":"2024-03-04T16:02:30.058990+0000","last_in_change":"2024-03-04T16:25:40.918436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:00:32.201931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-04T16:02:41.373159+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-04T16:02:42.125847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-04T16:02:44.027394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-04T16:02:46.000786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-04T16:02:48.076921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"99ecf38d-7bc9-412d-a194-f3d9eed8e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6801","nonce":1080993770}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6802","nonce":1080993770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6804","nonce":1080993770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6803","nonce":1080993770}]},"public_addr":"172.21.15.38:6801/1080993770","cluster_addr":"172.21.15.38:6802/1080993770","heartbeat_back_addr":"172.21.15.38:6804/1080993770","heartbeat_front_addr":"172.21.15.38:6803/1080993770","state":["exists","up"]},{"osd":1,"uuid":"f8c66eb4-4966-45ce-af41-b6d01713eb51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6805","nonce":2331598100}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6806","nonce":2331598100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6808","nonce":2331598100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6807","nonce":2331598100}]},"public_addr":"172.21.15.38:6805/2331598100","cluster_addr":"172.21.15.38:6806/2331598100","heartbeat_back_addr":"172.21.15.38:6808/2331598100","heartbeat_front_addr":"172.21.15.38:6807/2331598100","state":["exists","up"]},{"osd":2,"uuid":"afb2dd7e-d27a-4536-bb80-e5724b39ffd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6809","nonce":1898914368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6810","nonce":1898914368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6812","nonce":1898914368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6811","nonce":1898914368}]},"public_addr":"172.21.15.38:6809/1898914368","cluster_addr":"172.21.15.38:6810/1898914368","heartbeat_back_addr":"172.21.15.38:6812/1898914368","heartbeat_front_addr":"172.21.15.38:6811/1898914368","state":["exists","up"]},{"osd":3,"uuid":"13f08b82-22d5-4466-9ad4-27ffa15ed7c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6813","nonce":552035408}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6814","nonce":552035408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6816","nonce":552035408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6815","nonce":552035408}]},"public_addr":"172.21.15.38:6813/552035408","cluster_addr":"172.21.15.38:6814/552035408","heartbeat_back_addr":"172.21.15.38:6816/552035408","heartbeat_front_addr":"172.21.15.38:6815/552035408","state":["exists","up"]},{"osd":4,"uuid":"a58d697b-e1de-44cc-b791-8a8d68fd01f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6800","nonce":3997489718}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6801","nonce":3997489718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6803","nonce":3997489718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6802","nonce":3997489718}]},"public_addr":"172.21.15.176:6800/3997489718","cluster_addr":"172.21.15.176:6801/3997489718","heartbeat_back_addr":"172.21.15.176:6803/3997489718","heartbeat_front_addr":"172.21.15.176:6802/3997489718","state":["exists","up"]},{"osd":5,"uuid":"7adb17cc-4843-401e-a7fd-b726c4a99a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6804","nonce":1267814386}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6805","nonce":1267814386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6807","nonce":1267814386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6806","nonce":1267814386}]},"public_addr":"172.21.15.176:6804/1267814386","cluster_addr":"172.21.15.176:6805/1267814386","heartbeat_back_addr":"172.21.15.176:6807/1267814386","heartbeat_front_addr":"172.21.15.176:6806/1267814386","state":["exists","up"]},{"osd":6,"uuid":"0e12d14b-c02d-4acf-9784-8680f8ee40f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6808","nonce":587062426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6809","nonce":587062426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6811","nonce":587062426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6810","nonce":587062426}]},"public_addr":"172.21.15.176:6808/587062426","cluster_addr":"172.21.15.176:6809/587062426","heartbeat_back_addr":"172.21.15.176:6811/587062426","heartbeat_front_addr":"172.21.15.176:6810/587062426","state":["exists","up"]},{"osd":7,"uuid":"1f348507-1649-4e25-b5f5-cb67c6cd045e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6812","nonce":1019151537}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6813","nonce":1019151537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6815","nonce":1019151537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6814","nonce":1019151537}]},"public_addr":"172.21.15.176:6812/1019151537","cluster_addr":"172.21.15.176:6813/1019151537","heartbeat_back_addr":"172.21.15.176:6815/1019151537","heartbeat_front_addr":"172.21.15.176:6814/1019151537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:10.883038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:14.084945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:13.051432+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:21.496260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:06.333430+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:05.307099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:25:33.094823+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:37.230890+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.b","osds":[0,5,1]}],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":0,"to":5}]},{"pgid":"3.1c","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"5.13","mappings":[{"from":0,"to":5}]},{"pgid":"5.15","mappings":[{"from":1,"to":2}]},{"pgid":"6.15","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3142726897":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/452578506":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/501976220":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/1665273468":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/500449596":"2024-03-05T16:03:21.264346+0000","172.21.15.38:6800/4073253154":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/175394725":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/1249306019":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/2156780629":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/1997350354":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2476964715":"2024-03-05T15:58:15.846725+0000","172.21.15.38:0/1178105870":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/3751720132":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2774521881":"2024-03-05T15:58:15.846725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:26:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:46 smithi176 ceph-mon[33619]: 3.e scrub starts 2024-03-04T16:26:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:46 smithi176 ceph-mon[33619]: 3.e scrub ok 2024-03-04T16:26:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:46 smithi176 ceph-mon[33619]: 4.5 scrub starts 2024-03-04T16:26:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:46 smithi176 ceph-mon[33619]: 4.5 scrub ok 2024-03-04T16:26:46.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:46 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2427327601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:46.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[29779]: 3.e scrub starts 2024-03-04T16:26:46.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[29779]: 3.e scrub ok 2024-03-04T16:26:46.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[29779]: 4.5 scrub starts 2024-03-04T16:26:46.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[29779]: 4.5 scrub ok 2024-03-04T16:26:46.475 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2427327601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:46.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[38113]: 3.e scrub starts 2024-03-04T16:26:46.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[38113]: 3.e scrub ok 2024-03-04T16:26:46.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[38113]: 4.5 scrub starts 2024-03-04T16:26:46.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[38113]: 4.5 scrub ok 2024-03-04T16:26:46.476 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:46 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2427327601' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:46.711 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-04T16:26:46.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-04T16:26:46.966 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[29779]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[29779]: 6.1 scrub starts 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[29779]: 6.1 scrub ok 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[38113]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[38113]: 6.1 scrub starts 2024-03-04T16:26:47.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:47 smithi038 ceph-mon[38113]: 6.1 scrub ok 2024-03-04T16:26:47.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:47 smithi176 ceph-mon[33619]: pgmap v1141: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 654 B/s rd, 0 op/s 2024-03-04T16:26:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:47 smithi176 ceph-mon[33619]: 6.1 scrub starts 2024-03-04T16:26:47.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:47 smithi176 ceph-mon[33619]: 6.1 scrub ok 2024-03-04T16:26:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:48 smithi038 ceph-mon[29779]: 5.11 scrub starts 2024-03-04T16:26:48.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:48 smithi038 ceph-mon[29779]: 5.11 scrub ok 2024-03-04T16:26:48.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:48 smithi038 ceph-mon[38113]: 5.11 scrub starts 2024-03-04T16:26:48.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:48 smithi038 ceph-mon[38113]: 5.11 scrub ok 2024-03-04T16:26:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:48 smithi176 ceph-mon[33619]: 5.11 scrub starts 2024-03-04T16:26:48.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:48 smithi176 ceph-mon[33619]: 5.11 scrub ok 2024-03-04T16:26:48.959 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:26:48.959 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":516,"fsid":"c453baf0-da3f-11ee-95c6-87774f69a715","created":"2024-03-04T15:57:15.255572+0000","modified":"2024-03-04T16:26:43.023067+0000","last_up_change":"2024-03-04T16:02:30.058990+0000","last_in_change":"2024-03-04T16:25:40.918436+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-04T16:00:32.201931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"279","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-04T16:02:41.373159+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-04T16:02:42.125847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-04T16:02:44.027394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-04T16:02:46.000786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-04T16:02:48.076921+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"99ecf38d-7bc9-412d-a194-f3d9eed8e244","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6801","nonce":1080993770}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6802","nonce":1080993770}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6804","nonce":1080993770}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6803","nonce":1080993770}]},"public_addr":"172.21.15.38:6801/1080993770","cluster_addr":"172.21.15.38:6802/1080993770","heartbeat_back_addr":"172.21.15.38:6804/1080993770","heartbeat_front_addr":"172.21.15.38:6803/1080993770","state":["exists","up"]},{"osd":1,"uuid":"f8c66eb4-4966-45ce-af41-b6d01713eb51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6805","nonce":2331598100}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6806","nonce":2331598100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6808","nonce":2331598100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6807","nonce":2331598100}]},"public_addr":"172.21.15.38:6805/2331598100","cluster_addr":"172.21.15.38:6806/2331598100","heartbeat_back_addr":"172.21.15.38:6808/2331598100","heartbeat_front_addr":"172.21.15.38:6807/2331598100","state":["exists","up"]},{"osd":2,"uuid":"afb2dd7e-d27a-4536-bb80-e5724b39ffd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6809","nonce":1898914368}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6810","nonce":1898914368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6812","nonce":1898914368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6811","nonce":1898914368}]},"public_addr":"172.21.15.38:6809/1898914368","cluster_addr":"172.21.15.38:6810/1898914368","heartbeat_back_addr":"172.21.15.38:6812/1898914368","heartbeat_front_addr":"172.21.15.38:6811/1898914368","state":["exists","up"]},{"osd":3,"uuid":"13f08b82-22d5-4466-9ad4-27ffa15ed7c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6813","nonce":552035408}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6814","nonce":552035408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6816","nonce":552035408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.38:6815","nonce":552035408}]},"public_addr":"172.21.15.38:6813/552035408","cluster_addr":"172.21.15.38:6814/552035408","heartbeat_back_addr":"172.21.15.38:6816/552035408","heartbeat_front_addr":"172.21.15.38:6815/552035408","state":["exists","up"]},{"osd":4,"uuid":"a58d697b-e1de-44cc-b791-8a8d68fd01f3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6800","nonce":3997489718}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6801","nonce":3997489718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6803","nonce":3997489718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6802","nonce":3997489718}]},"public_addr":"172.21.15.176:6800/3997489718","cluster_addr":"172.21.15.176:6801/3997489718","heartbeat_back_addr":"172.21.15.176:6803/3997489718","heartbeat_front_addr":"172.21.15.176:6802/3997489718","state":["exists","up"]},{"osd":5,"uuid":"7adb17cc-4843-401e-a7fd-b726c4a99a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6804","nonce":1267814386}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6805","nonce":1267814386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6807","nonce":1267814386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6806","nonce":1267814386}]},"public_addr":"172.21.15.176:6804/1267814386","cluster_addr":"172.21.15.176:6805/1267814386","heartbeat_back_addr":"172.21.15.176:6807/1267814386","heartbeat_front_addr":"172.21.15.176:6806/1267814386","state":["exists","up"]},{"osd":6,"uuid":"0e12d14b-c02d-4acf-9784-8680f8ee40f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6808","nonce":587062426}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6809","nonce":587062426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6811","nonce":587062426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6810","nonce":587062426}]},"public_addr":"172.21.15.176:6808/587062426","cluster_addr":"172.21.15.176:6809/587062426","heartbeat_back_addr":"172.21.15.176:6811/587062426","heartbeat_front_addr":"172.21.15.176:6810/587062426","state":["exists","up"]},{"osd":7,"uuid":"1f348507-1649-4e25-b5f5-cb67c6cd045e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":509,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6812","nonce":1019151537}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6813","nonce":1019151537}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6815","nonce":1019151537}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.176:6814","nonce":1019151537}]},"public_addr":"172.21.15.176:6812/1019151537","cluster_addr":"172.21.15.176:6813/1019151537","heartbeat_back_addr":"172.21.15.176:6815/1019151537","heartbeat_front_addr":"172.21.15.176:6814/1019151537","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:10.883038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:14.084945+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:13.051432+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:21.496260+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:06.333430+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:05.307099+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:25:33.094823+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-04T16:26:37.230890+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.b","osds":[0,5,1]}],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":0,"to":5}]},{"pgid":"3.1c","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"5.13","mappings":[{"from":0,"to":5}]},{"pgid":"5.15","mappings":[{"from":1,"to":2}]},{"pgid":"6.15","mappings":[{"from":7,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3142726897":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/452578506":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/501976220":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/1665273468":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/500449596":"2024-03-05T16:03:21.264346+0000","172.21.15.38:6800/4073253154":"2024-03-05T16:03:21.264346+0000","172.21.15.38:0/175394725":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/1249306019":"2024-03-05T15:58:15.846725+0000","172.21.15.38:6800/2156780629":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/1997350354":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2476964715":"2024-03-05T15:58:15.846725+0000","172.21.15.38:0/1178105870":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/3751720132":"2024-03-05T15:57:48.640902+0000","172.21.15.38:0/2774521881":"2024-03-05T15:58:15.846725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-04T16:26:49.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-04T16:26:49.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-04T16:26:49.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-04T16:26:49.685 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-04T16:26:49.686 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-04T16:26:49.686 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-04T16:26:49.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-04T16:26:49.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-04T16:26:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 4.1c scrub starts 2024-03-04T16:26:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 4.1c scrub ok 2024-03-04T16:26:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 580 B/s rd, 0 op/s 2024-03-04T16:26:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 5.c deep-scrub starts 2024-03-04T16:26:49.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 5.c deep-scrub ok 2024-03-04T16:26:49.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 4.16 scrub starts 2024-03-04T16:26:49.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: 4.16 scrub ok 2024-03-04T16:26:49.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:49 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/116213425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:49.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 4.1c scrub starts 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 4.1c scrub ok 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 580 B/s rd, 0 op/s 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 5.c deep-scrub starts 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 5.c deep-scrub ok 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 4.16 scrub starts 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: 4.16 scrub ok 2024-03-04T16:26:49.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/116213425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 4.1c scrub starts 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 4.1c scrub ok 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 580 B/s rd, 0 op/s 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 5.c deep-scrub starts 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 5.c deep-scrub ok 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 4.16 scrub starts 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: 4.16 scrub ok 2024-03-04T16:26:49.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:49 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/116213425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-04T16:26:50.301 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.302 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.302 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.303 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.304 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.306 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.306 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.325 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:26:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 3.7 scrub starts 2024-03-04T16:26:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 3.7 scrub ok 2024-03-04T16:26:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 6.1d scrub starts 2024-03-04T16:26:50.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 3.c scrub starts 2024-03-04T16:26:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 3.c scrub ok 2024-03-04T16:26:50.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[38113]: 6.1d scrub ok 2024-03-04T16:26:50.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 3.7 scrub starts 2024-03-04T16:26:50.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 3.7 scrub ok 2024-03-04T16:26:50.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 6.1d scrub starts 2024-03-04T16:26:50.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:50.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 3.c scrub starts 2024-03-04T16:26:50.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 3.c scrub ok 2024-03-04T16:26:50.746 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:50 smithi038 ceph-mon[29779]: 6.1d scrub ok 2024-03-04T16:26:50.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 3.7 scrub starts 2024-03-04T16:26:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 3.7 scrub ok 2024-03-04T16:26:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 6.1d scrub starts 2024-03-04T16:26:50.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:50.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 3.c scrub starts 2024-03-04T16:26:50.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 3.c scrub ok 2024-03-04T16:26:50.860 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:50 smithi176 ceph-mon[33619]: 6.1d scrub ok 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:26:51.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:26:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:26:51.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:51 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:26:51.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:26:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:26:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:51 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:26:51.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:51 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:26:52.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[29779]: 5.18 scrub starts 2024-03-04T16:26:52.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[29779]: 5.18 scrub ok 2024-03-04T16:26:52.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[29779]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:52.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[29779]: 6.7 scrub starts 2024-03-04T16:26:52.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[38113]: 5.18 scrub starts 2024-03-04T16:26:52.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[38113]: 5.18 scrub ok 2024-03-04T16:26:52.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[38113]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:52.685 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:52 smithi038 ceph-mon[38113]: 6.7 scrub starts 2024-03-04T16:26:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:52 smithi176 ceph-mon[33619]: 5.18 scrub starts 2024-03-04T16:26:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:52 smithi176 ceph-mon[33619]: 5.18 scrub ok 2024-03-04T16:26:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:52 smithi176 ceph-mon[33619]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:52.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:52 smithi176 ceph-mon[33619]: 6.7 scrub starts 2024-03-04T16:26:53.492 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:26:53] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:26:53.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:26:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-mon[33619]: 6.1e deep-scrub starts 2024-03-04T16:26:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-mon[33619]: 6.1e deep-scrub ok 2024-03-04T16:26:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-mon[33619]: 6.7 scrub ok 2024-03-04T16:26:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-mon[33619]: 4.f scrub starts 2024-03-04T16:26:53.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:53 smithi176 ceph-mon[33619]: 4.f scrub ok 2024-03-04T16:26:53.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[29779]: 6.1e deep-scrub starts 2024-03-04T16:26:53.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[29779]: 6.1e deep-scrub ok 2024-03-04T16:26:53.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[29779]: 6.7 scrub ok 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[29779]: 4.f scrub starts 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[29779]: 4.f scrub ok 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[38113]: 6.1e deep-scrub starts 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[38113]: 6.1e deep-scrub ok 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[38113]: 6.7 scrub ok 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[38113]: 4.f scrub starts 2024-03-04T16:26:53.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:53 smithi038 ceph-mon[38113]: 4.f scrub ok 2024-03-04T16:26:54.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: 5.8 scrub starts 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: 5.8 scrub ok 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: 4.12 scrub starts 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: 4.12 scrub ok 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: 5.8 scrub starts 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: 5.8 scrub ok 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:54.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: 4.12 scrub starts 2024-03-04T16:26:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: 4.12 scrub ok 2024-03-04T16:26:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-04T16:26:54.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:54 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:54.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: 5.8 scrub starts 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: 5.8 scrub ok 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: 4.12 scrub starts 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: 4.12 scrub ok 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-04T16:26:54.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:54 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:26:55.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[29779]: 4.18 scrub starts 2024-03-04T16:26:55.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[29779]: 4.18 scrub ok 2024-03-04T16:26:55.672 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[29779]: 5.5 scrub starts 2024-03-04T16:26:55.673 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[29779]: 5.5 scrub ok 2024-03-04T16:26:55.673 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[38113]: 4.18 scrub starts 2024-03-04T16:26:55.673 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[38113]: 4.18 scrub ok 2024-03-04T16:26:55.673 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[38113]: 5.5 scrub starts 2024-03-04T16:26:55.673 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:55 smithi038 ceph-mon[38113]: 5.5 scrub ok 2024-03-04T16:26:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:55 smithi176 ceph-mon[33619]: 4.18 scrub starts 2024-03-04T16:26:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:55 smithi176 ceph-mon[33619]: 4.18 scrub ok 2024-03-04T16:26:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:55 smithi176 ceph-mon[33619]: 5.5 scrub starts 2024-03-04T16:26:55.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:55 smithi176 ceph-mon[33619]: 5.5 scrub ok 2024-03-04T16:26:56.773 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: 4.1d scrub starts 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: 4.1d scrub ok 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: osdmap e517: 8 total, 8 up, 8 in 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: 5.1d scrub starts 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: 5.1d scrub ok 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[29779]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: 4.1d scrub starts 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: 4.1d scrub ok 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:26:56.774 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:26:56.775 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: osdmap e517: 8 total, 8 up, 8 in 2024-03-04T16:26:56.775 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: 5.1d scrub starts 2024-03-04T16:26:56.775 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: 5.1d scrub ok 2024-03-04T16:26:56.775 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:56 smithi038 ceph-mon[38113]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:56.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: 4.1d scrub starts 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: 4.1d scrub ok 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: osdmap e517: 8 total, 8 up, 8 in 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: 5.1d scrub starts 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: 5.1d scrub ok 2024-03-04T16:26:56.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:56 smithi176 ceph-mon[33619]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.e scrub starts 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.e scrub ok 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.0 deep-scrub starts 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.0 deep-scrub ok 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 3.9 deep-scrub starts 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 3.9 deep-scrub ok 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.1f deep-scrub starts 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.1f deep-scrub ok 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.1a scrub starts 2024-03-04T16:26:57.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[29779]: 5.1a scrub ok 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.e scrub starts 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.e scrub ok 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.0 deep-scrub starts 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.0 deep-scrub ok 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 3.9 deep-scrub starts 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 3.9 deep-scrub ok 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.1f deep-scrub starts 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.1f deep-scrub ok 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.1a scrub starts 2024-03-04T16:26:57.809 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:57 smithi038 ceph-mon[38113]: 5.1a scrub ok 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.e scrub starts 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.e scrub ok 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.0 deep-scrub starts 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.0 deep-scrub ok 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 3.9 deep-scrub starts 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 3.9 deep-scrub ok 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.1f deep-scrub starts 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.1f deep-scrub ok 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.1a scrub starts 2024-03-04T16:26:57.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:57 smithi176 ceph-mon[33619]: 5.1a scrub ok 2024-03-04T16:26:58.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[38113]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[38113]: 3.1f scrub starts 2024-03-04T16:26:58.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[38113]: 3.1f scrub ok 2024-03-04T16:26:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[29779]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[29779]: 3.1f scrub starts 2024-03-04T16:26:58.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:58 smithi038 ceph-mon[29779]: 3.1f scrub ok 2024-03-04T16:26:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:58 smithi176 ceph-mon[33619]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:26:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:58 smithi176 ceph-mon[33619]: 3.1f scrub starts 2024-03-04T16:26:58.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:58 smithi176 ceph-mon[33619]: 3.1f scrub ok 2024-03-04T16:26:59.220 INFO:teuthology.orchestra.run.smithi038.stdout:253403070759 2024-03-04T16:26:59.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-04T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:59 smithi176 ceph-mon[33619]: 5.1 deep-scrub starts 2024-03-04T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:59 smithi176 ceph-mon[33619]: 5.1 deep-scrub ok 2024-03-04T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:59 smithi176 ceph-mon[33619]: 4.15 scrub starts 2024-03-04T16:26:59.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:26:59 smithi176 ceph-mon[33619]: 4.15 scrub ok 2024-03-04T16:26:59.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[29779]: 5.1 deep-scrub starts 2024-03-04T16:26:59.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[29779]: 5.1 deep-scrub ok 2024-03-04T16:26:59.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[29779]: 4.15 scrub starts 2024-03-04T16:26:59.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[29779]: 4.15 scrub ok 2024-03-04T16:26:59.981 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[38113]: 5.1 deep-scrub starts 2024-03-04T16:26:59.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[38113]: 5.1 deep-scrub ok 2024-03-04T16:26:59.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[38113]: 4.15 scrub starts 2024-03-04T16:26:59.982 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:26:59 smithi038 ceph-mon[38113]: 4.15 scrub ok 2024-03-04T16:27:00.365 INFO:teuthology.orchestra.run.smithi038.stdout:111669150009 2024-03-04T16:27:00.366 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-04T16:27:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:00 smithi176 ceph-mon[33619]: 6.f scrub starts 2024-03-04T16:27:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:00 smithi176 ceph-mon[33619]: 6.f scrub ok 2024-03-04T16:27:00.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:00 smithi176 ceph-mon[33619]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[29779]: 6.f scrub starts 2024-03-04T16:27:00.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[29779]: 6.f scrub ok 2024-03-04T16:27:00.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[29779]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[38113]: 6.f scrub starts 2024-03-04T16:27:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[38113]: 6.f scrub ok 2024-03-04T16:27:00.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:00 smithi038 ceph-mon[38113]: pgmap v1149: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:01.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:01 smithi038 ceph-mon[29779]: 3.10 scrub starts 2024-03-04T16:27:01.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:01 smithi038 ceph-mon[29779]: 3.10 scrub ok 2024-03-04T16:27:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:01 smithi038 ceph-mon[38113]: 3.10 scrub starts 2024-03-04T16:27:01.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:01 smithi038 ceph-mon[38113]: 3.10 scrub ok 2024-03-04T16:27:01.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:01 smithi176 ceph-mon[33619]: 3.10 scrub starts 2024-03-04T16:27:01.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:01 smithi176 ceph-mon[33619]: 3.10 scrub ok 2024-03-04T16:27:02.145 INFO:teuthology.orchestra.run.smithi038.stdout:77309411648 2024-03-04T16:27:02.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-04T16:27:02.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[29779]: 4.1 scrub starts 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[29779]: 4.1 scrub ok 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[29779]: 4.17 scrub starts 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[29779]: 4.17 scrub ok 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[29779]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[38113]: 4.1 scrub starts 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[38113]: 4.1 scrub ok 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[38113]: 4.17 scrub starts 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[38113]: 4.17 scrub ok 2024-03-04T16:27:02.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:02 smithi038 ceph-mon[38113]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:03.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:02 smithi176 ceph-mon[33619]: 4.1 scrub starts 2024-03-04T16:27:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:02 smithi176 ceph-mon[33619]: 4.1 scrub ok 2024-03-04T16:27:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:02 smithi176 ceph-mon[33619]: 4.17 scrub starts 2024-03-04T16:27:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:02 smithi176 ceph-mon[33619]: 4.17 scrub ok 2024-03-04T16:27:03.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:02 smithi176 ceph-mon[33619]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:03.618 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:03 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:27:03] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:27:03.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:03 smithi038 ceph-mon[38113]: 6.3 scrub starts 2024-03-04T16:27:03.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:03 smithi038 ceph-mon[38113]: 6.3 scrub ok 2024-03-04T16:27:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:03 smithi038 ceph-mon[29779]: 6.3 scrub starts 2024-03-04T16:27:03.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:03 smithi038 ceph-mon[29779]: 6.3 scrub ok 2024-03-04T16:27:04.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:03 smithi176 ceph-mon[33619]: 6.3 scrub starts 2024-03-04T16:27:04.009 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:03 smithi176 ceph-mon[33619]: 6.3 scrub ok 2024-03-04T16:27:04.009 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:27:03 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:27:04.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:04 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:04.881 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:04 smithi038 ceph-mon[29779]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:04.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:04 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:04.881 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:04 smithi038 ceph-mon[38113]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:04 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:05.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:04 smithi176 ceph-mon[33619]: pgmap v1151: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:05.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[38113]: 6.1a deep-scrub starts 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[38113]: 6.1a deep-scrub ok 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[38113]: 3.13 deep-scrub starts 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[38113]: 3.13 deep-scrub ok 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[29779]: 6.1a deep-scrub starts 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[29779]: 6.1a deep-scrub ok 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[29779]: 3.13 deep-scrub starts 2024-03-04T16:27:05.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:05 smithi038 ceph-mon[29779]: 3.13 deep-scrub ok 2024-03-04T16:27:05.956 INFO:teuthology.orchestra.run.smithi038.stdout:137438953783 2024-03-04T16:27:05.957 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-04T16:27:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:05 smithi176 ceph-mon[33619]: 6.1a deep-scrub starts 2024-03-04T16:27:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:05 smithi176 ceph-mon[33619]: 6.1a deep-scrub ok 2024-03-04T16:27:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:05 smithi176 ceph-mon[33619]: 3.13 deep-scrub starts 2024-03-04T16:27:06.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:05 smithi176 ceph-mon[33619]: 3.13 deep-scrub ok 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 4.9 scrub starts 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 4.9 scrub ok 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 6.15 scrub starts 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 6.16 deep-scrub starts 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 6.15 scrub ok 2024-03-04T16:27:06.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[29779]: 6.16 deep-scrub ok 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 4.9 scrub starts 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 4.9 scrub ok 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 6.15 scrub starts 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 6.16 deep-scrub starts 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 6.15 scrub ok 2024-03-04T16:27:06.946 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:06 smithi038 ceph-mon[38113]: 6.16 deep-scrub ok 2024-03-04T16:27:06.973 INFO:teuthology.orchestra.run.smithi038.stdout:55834575174 2024-03-04T16:27:06.973 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-04T16:27:07.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 4.9 scrub starts 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 4.9 scrub ok 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 6.15 scrub starts 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 6.16 deep-scrub starts 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 6.15 scrub ok 2024-03-04T16:27:07.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:06 smithi176 ceph-mon[33619]: 6.16 deep-scrub ok 2024-03-04T16:27:07.601 INFO:teuthology.orchestra.run.smithi038.stdout:180388626738 2024-03-04T16:27:07.601 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-04T16:27:07.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 4.a scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 4.a scrub ok 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 3.19 scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 3.19 scrub ok 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 5.f deep-scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[38113]: 5.f deep-scrub ok 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 4.a scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 4.a scrub ok 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 3.19 scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 3.19 scrub ok 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 5.f deep-scrub starts 2024-03-04T16:27:07.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:07 smithi038 ceph-mon[29779]: 5.f deep-scrub ok 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 4.a scrub starts 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 4.a scrub ok 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 3.19 scrub starts 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 3.19 scrub ok 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 5.f deep-scrub starts 2024-03-04T16:27:08.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:07 smithi176 ceph-mon[33619]: 5.f deep-scrub ok 2024-03-04T16:27:08.405 INFO:teuthology.orchestra.run.smithi038.stdout:34359738699 2024-03-04T16:27:08.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-04T16:27:08.924 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:08.925 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:08.968 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[29779]: 4.0 scrub starts 2024-03-04T16:27:08.968 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[29779]: 4.0 scrub ok 2024-03-04T16:27:08.968 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[29779]: 5.19 scrub starts 2024-03-04T16:27:08.968 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[29779]: 5.19 scrub ok 2024-03-04T16:27:08.969 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[29779]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:27:08.969 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[38113]: 4.0 scrub starts 2024-03-04T16:27:08.969 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[38113]: 4.0 scrub ok 2024-03-04T16:27:08.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[38113]: 5.19 scrub starts 2024-03-04T16:27:08.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[38113]: 5.19 scrub ok 2024-03-04T16:27:08.970 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:08 smithi038 ceph-mon[38113]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:27:08.971 INFO:teuthology.orchestra.run.smithi038.stdout:214748365101 2024-03-04T16:27:08.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-04T16:27:08.983 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:08.983 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:09.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:08 smithi176 ceph-mon[33619]: 4.0 scrub starts 2024-03-04T16:27:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:08 smithi176 ceph-mon[33619]: 4.0 scrub ok 2024-03-04T16:27:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:08 smithi176 ceph-mon[33619]: 5.19 scrub starts 2024-03-04T16:27:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:08 smithi176 ceph-mon[33619]: 5.19 scrub ok 2024-03-04T16:27:09.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:08 smithi176 ceph-mon[33619]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-04T16:27:09.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 6.e scrub starts 2024-03-04T16:27:09.864 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 6.e scrub ok 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 3.5 scrub starts 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 3.5 scrub ok 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: osdmap e518: 8 total, 8 up, 8 in 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 4.6 scrub starts 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 4.6 scrub ok 2024-03-04T16:27:09.865 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[29779]: 5.16 scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 6.e scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 6.e scrub ok 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 3.5 scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 3.5 scrub ok 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: osdmap e518: 8 total, 8 up, 8 in 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 4.6 scrub starts 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 4.6 scrub ok 2024-03-04T16:27:09.866 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:09 smithi038 ceph-mon[38113]: 5.16 scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 6.e scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 6.e scrub ok 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 3.5 scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 3.5 scrub ok 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: osdmap e518: 8 total, 8 up, 8 in 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 4.6 scrub starts 2024-03-04T16:27:10.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:10.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 4.6 scrub ok 2024-03-04T16:27:10.109 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:09 smithi176 ceph-mon[33619]: 5.16 scrub starts 2024-03-04T16:27:10.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: 4.1a scrub starts 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: 4.1a scrub ok 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: 6.b scrub starts 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: 5.16 scrub ok 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: 6.b scrub ok 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[29779]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: 4.1a scrub starts 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: 4.1a scrub ok 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: 6.b scrub starts 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: 5.16 scrub ok 2024-03-04T16:27:10.976 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: 6.b scrub ok 2024-03-04T16:27:10.977 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:10 smithi038 ceph-mon[38113]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:11.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: 4.1a scrub starts 2024-03-04T16:27:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: 4.1a scrub ok 2024-03-04T16:27:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: 6.b scrub starts 2024-03-04T16:27:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: 5.16 scrub ok 2024-03-04T16:27:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: 6.b scrub ok 2024-03-04T16:27:11.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:10 smithi176 ceph-mon[33619]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:11.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 6.1b scrub starts 2024-03-04T16:27:11.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 4.8 scrub starts 2024-03-04T16:27:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 4.8 scrub ok 2024-03-04T16:27:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 6.1b scrub ok 2024-03-04T16:27:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 4.10 scrub starts 2024-03-04T16:27:11.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[29779]: 4.10 scrub ok 2024-03-04T16:27:11.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 6.1b scrub starts 2024-03-04T16:27:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 4.8 scrub starts 2024-03-04T16:27:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 4.8 scrub ok 2024-03-04T16:27:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 6.1b scrub ok 2024-03-04T16:27:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 4.10 scrub starts 2024-03-04T16:27:11.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:11 smithi038 ceph-mon[38113]: 4.10 scrub ok 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 6.1b scrub starts 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 4.8 scrub starts 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 4.8 scrub ok 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 6.1b scrub ok 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 4.10 scrub starts 2024-03-04T16:27:12.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:11 smithi176 ceph-mon[33619]: 4.10 scrub ok 2024-03-04T16:27:12.992 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:12 smithi038 ceph-mon[38113]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:12.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:12 smithi038 ceph-mon[29779]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:13.107 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:12 smithi176 ceph-mon[33619]: pgmap v1156: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:13.359 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:27:13] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:27:13.961 INFO:teuthology.orchestra.run.smithi038.stdout:77309411651 2024-03-04T16:27:14.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:27:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:27:14.190 INFO:teuthology.orchestra.run.smithi038.stdout:137438953785 2024-03-04T16:27:14.689 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411648 got 77309411651 for osd.2 2024-03-04T16:27:14.689 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:14.692 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:14.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: osdmap e519: 8 total, 8 up, 8 in 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: 6.11 deep-scrub starts 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: 6.11 deep-scrub ok 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1508735020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1393654170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:14.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: osdmap e519: 8 total, 8 up, 8 in 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: 6.11 deep-scrub starts 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: 6.11 deep-scrub ok 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1508735020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:27:14.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:14 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1393654170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: osdmap e519: 8 total, 8 up, 8 in 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: 6.11 deep-scrub starts 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: 6.11 deep-scrub ok 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1508735020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-04T16:27:15.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:14 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1393654170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-04T16:27:16.155 INFO:teuthology.orchestra.run.smithi038.stdout:253403070763 2024-03-04T16:27:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[29779]: 3.1c scrub starts 2024-03-04T16:27:16.155 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[29779]: 3.1c scrub ok 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[29779]: 3.2 scrub starts 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[29779]: 3.2 scrub ok 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[38113]: 3.1c scrub starts 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[38113]: 3.1c scrub ok 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[38113]: 3.2 scrub starts 2024-03-04T16:27:16.156 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:15 smithi038 ceph-mon[38113]: 3.2 scrub ok 2024-03-04T16:27:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:15 smithi176 ceph-mon[33619]: 3.1c scrub starts 2024-03-04T16:27:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:15 smithi176 ceph-mon[33619]: 3.1c scrub ok 2024-03-04T16:27:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:15 smithi176 ceph-mon[33619]: 3.2 scrub starts 2024-03-04T16:27:16.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:15 smithi176 ceph-mon[33619]: 3.2 scrub ok 2024-03-04T16:27:16.366 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953783 got 137438953785 for osd.4 2024-03-04T16:27:16.366 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:16.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[29779]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:16.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[29779]: 3.11 scrub starts 2024-03-04T16:27:16.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[29779]: 3.11 scrub ok 2024-03-04T16:27:16.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1879341010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:27:16.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[38113]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:16.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[38113]: 3.11 scrub starts 2024-03-04T16:27:16.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[38113]: 3.11 scrub ok 2024-03-04T16:27:16.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:16 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1879341010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:27:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:16 smithi176 ceph-mon[33619]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:16 smithi176 ceph-mon[33619]: 3.11 scrub starts 2024-03-04T16:27:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:16 smithi176 ceph-mon[33619]: 3.11 scrub ok 2024-03-04T16:27:17.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:16 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1879341010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-04T16:27:18.184 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070759 got 253403070763 for osd.7 2024-03-04T16:27:18.184 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:19.135 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 5.2 scrub starts 2024-03-04T16:27:19.135 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 5.2 scrub ok 2024-03-04T16:27:19.135 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 4.1b scrub starts 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 4.1b scrub ok 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 3.f scrub starts 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[29779]: 3.f scrub ok 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 5.2 scrub starts 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 5.2 scrub ok 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 4.1b scrub starts 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 4.1b scrub ok 2024-03-04T16:27:19.136 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 3.f scrub starts 2024-03-04T16:27:19.137 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:18 smithi038 ceph-mon[38113]: 3.f scrub ok 2024-03-04T16:27:19.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 5.2 scrub starts 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 5.2 scrub ok 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 560 B/s rd, 0 op/s 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 4.1b scrub starts 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 4.1b scrub ok 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 3.f scrub starts 2024-03-04T16:27:19.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:18 smithi176 ceph-mon[33619]: 3.f scrub ok 2024-03-04T16:27:20.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[29779]: 2.2 scrub starts 2024-03-04T16:27:20.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[29779]: 2.2 scrub ok 2024-03-04T16:27:20.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[29779]: 5.a scrub starts 2024-03-04T16:27:20.127 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[29779]: 5.a scrub ok 2024-03-04T16:27:20.127 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[38113]: 2.2 scrub starts 2024-03-04T16:27:20.128 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[38113]: 2.2 scrub ok 2024-03-04T16:27:20.128 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[38113]: 5.a scrub starts 2024-03-04T16:27:20.128 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:19 smithi038 ceph-mon[38113]: 5.a scrub ok 2024-03-04T16:27:20.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:19 smithi176 ceph-mon[33619]: 2.2 scrub starts 2024-03-04T16:27:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:19 smithi176 ceph-mon[33619]: 2.2 scrub ok 2024-03-04T16:27:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:19 smithi176 ceph-mon[33619]: 5.a scrub starts 2024-03-04T16:27:20.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:19 smithi176 ceph-mon[33619]: 5.a scrub ok 2024-03-04T16:27:20.716 INFO:teuthology.orchestra.run.smithi038.stdout:55834575178 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 6.17 scrub starts 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 4.11 scrub starts 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 4.11 scrub ok 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 6.17 scrub ok 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 3.1d deep-scrub starts 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: 3.1d deep-scrub ok 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1512062574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:27:20.993 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 6.17 scrub starts 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 4.11 scrub starts 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 4.11 scrub ok 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 6.17 scrub ok 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 3.1d deep-scrub starts 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: 3.1d deep-scrub ok 2024-03-04T16:27:20.994 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:20 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1512062574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:27:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: pgmap v1161: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 6.17 scrub starts 2024-03-04T16:27:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 4.11 scrub starts 2024-03-04T16:27:21.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 4.11 scrub ok 2024-03-04T16:27:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 6.17 scrub ok 2024-03-04T16:27:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 3.1d deep-scrub starts 2024-03-04T16:27:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: 3.1d deep-scrub ok 2024-03-04T16:27:21.359 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:20 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1512062574' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-04T16:27:21.987 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575174 got 55834575178 for osd.1 2024-03-04T16:27:21.987 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:22.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:21 smithi038 ceph-mon[29779]: 6.19 scrub starts 2024-03-04T16:27:22.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:21 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/3027691383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:27:22.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:21 smithi038 ceph-mon[38113]: 6.19 scrub starts 2024-03-04T16:27:22.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:21 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/3027691383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:27:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:21 smithi176 ceph-mon[33619]: 6.19 scrub starts 2024-03-04T16:27:22.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:21 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/3027691383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-04T16:27:22.621 INFO:teuthology.orchestra.run.smithi038.stdout:111669150014 2024-03-04T16:27:23.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: 6.19 scrub ok 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: 5.9 scrub starts 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: 5.9 scrub ok 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: 4.14 scrub starts 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[29779]: 4.14 scrub ok 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: 6.19 scrub ok 2024-03-04T16:27:23.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:23.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: 5.9 scrub starts 2024-03-04T16:27:23.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: 5.9 scrub ok 2024-03-04T16:27:23.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: 4.14 scrub starts 2024-03-04T16:27:23.244 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:22 smithi038 ceph-mon[38113]: 4.14 scrub ok 2024-03-04T16:27:23.341 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150009 got 111669150014 for osd.3 2024-03-04T16:27:23.341 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: 6.19 scrub ok 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: 5.9 scrub starts 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: 5.9 scrub ok 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: 4.14 scrub starts 2024-03-04T16:27:23.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:22 smithi176 ceph-mon[33619]: 4.14 scrub ok 2024-03-04T16:27:23.560 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:27:23] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:27:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:23 smithi176 ceph-mon[33619]: 3.a scrub starts 2024-03-04T16:27:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:23 smithi176 ceph-mon[33619]: 3.a scrub ok 2024-03-04T16:27:24.108 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:23 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:24.108 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:27:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:27:24.242 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[29779]: 3.a scrub starts 2024-03-04T16:27:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[29779]: 3.a scrub ok 2024-03-04T16:27:24.243 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[38113]: 3.a scrub starts 2024-03-04T16:27:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[38113]: 3.a scrub ok 2024-03-04T16:27:24.243 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:23 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-04T16:27:25.190 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[29779]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[29779]: 5.4 scrub starts 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[29779]: 5.4 scrub ok 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[38113]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:25.191 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[38113]: 5.4 scrub starts 2024-03-04T16:27:25.192 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:24 smithi038 ceph-mon[38113]: 5.4 scrub ok 2024-03-04T16:27:25.200 INFO:teuthology.orchestra.run.smithi038.stdout:214748365105 2024-03-04T16:27:25.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:24 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:25.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:24 smithi176 ceph-mon[33619]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:25.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:24 smithi176 ceph-mon[33619]: 5.4 scrub starts 2024-03-04T16:27:25.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:24 smithi176 ceph-mon[33619]: 5.4 scrub ok 2024-03-04T16:27:25.444 INFO:teuthology.orchestra.run.smithi038.stdout:34359738702 2024-03-04T16:27:26.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[29779]: osdmap e520: 8 total, 8 up, 8 in 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/2796277008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1148587021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[38113]: osdmap e520: 8 total, 8 up, 8 in 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/2796277008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:27:26.090 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:25 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1148587021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:27:26.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:25 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:27:26.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:25 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:27:26.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:25 smithi176 ceph-mon[33619]: osdmap e520: 8 total, 8 up, 8 in 2024-03-04T16:27:26.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/2796277008' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-04T16:27:26.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:25 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1148587021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-04T16:27:26.405 INFO:teuthology.orchestra.run.smithi038.stdout:180388626743 2024-03-04T16:27:26.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365101 got 214748365105 for osd.6 2024-03-04T16:27:26.657 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: purged_snaps scrub starts 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: purged_snaps scrub ok 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: 3.1b scrub starts 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: 3.1b scrub ok 2024-03-04T16:27:27.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/1410673808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: osdmap e521: 8 total, 8 up, 8 in 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: 4.1f scrub starts 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[29779]: 4.1f scrub ok 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: purged_snaps scrub starts 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: purged_snaps scrub ok 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: 3.1b scrub starts 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: 3.1b scrub ok 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/1410673808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: osdmap e521: 8 total, 8 up, 8 in 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: 4.1f scrub starts 2024-03-04T16:27:27.240 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:26 smithi038 ceph-mon[38113]: 4.1f scrub ok 2024-03-04T16:27:27.268 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626738 got 180388626743 for osd.5 2024-03-04T16:27:27.269 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:27.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: purged_snaps scrub starts 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: purged_snaps scrub ok 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: 3.1b scrub starts 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: 3.1b scrub ok 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/1410673808' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: osdmap e521: 8 total, 8 up, 8 in 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: 4.1f scrub starts 2024-03-04T16:27:27.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:26 smithi176 ceph-mon[33619]: 4.1f scrub ok 2024-03-04T16:27:27.729 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738699 got 34359738702 for osd.0 2024-03-04T16:27:27.729 DEBUG:teuthology.parallel:result is None 2024-03-04T16:27:27.729 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-04T16:27:27.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:27:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:28.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:27 smithi038 ceph-mon[29779]: 2.0 scrub starts 2024-03-04T16:27:28.235 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:27 smithi038 ceph-mon[29779]: 2.0 scrub ok 2024-03-04T16:27:28.236 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:27 smithi038 ceph-mon[38113]: 2.0 scrub starts 2024-03-04T16:27:28.236 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:27 smithi038 ceph-mon[38113]: 2.0 scrub ok 2024-03-04T16:27:28.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:27 smithi176 ceph-mon[33619]: 2.0 scrub starts 2024-03-04T16:27:28.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:27 smithi176 ceph-mon[33619]: 2.0 scrub ok 2024-03-04T16:27:29.238 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:28 smithi038 ceph-mon[29779]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:29.238 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:28 smithi038 ceph-mon[38113]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:29.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:28 smithi176 ceph-mon[33619]: pgmap v1167: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:30.052 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:27:30.055 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 6.10 scrub starts 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 6.5 scrub starts 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 6.10 scrub ok 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 6.5 scrub ok 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 5.1b scrub starts 2024-03-04T16:27:30.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[29779]: 5.1b scrub ok 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 6.10 scrub starts 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 6.5 scrub starts 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 6.10 scrub ok 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 6.5 scrub ok 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 5.1b scrub starts 2024-03-04T16:27:30.064 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:29 smithi038 ceph-mon[38113]: 5.1b scrub ok 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 6.10 scrub starts 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 6.5 scrub starts 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 6.10 scrub ok 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 6.5 scrub ok 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 5.1b scrub starts 2024-03-04T16:27:30.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:29 smithi176 ceph-mon[33619]: 5.1b scrub ok 2024-03-04T16:27:30.771 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1168,"stamp":"2024-03-04T16:27:29.813567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3219,"num_read_kb":3065,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":4153671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1309,"ondisk_log_size":1309,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7092716,"kb_used_data":8324,"kb_used_omap":12,"kb_used_meta":7084275,"kb_avail":742868500,"statfs":{"total":767960285184,"available":760697344000,"internally_reserved":0,"allocated":8523776,"data_stored":5141044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7254298162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002315"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511729+0000","last_change":"2024-03-04T16:27:03.801816+0000","last_active":"2024-03-04T16:27:26.511729+0000","last_peered":"2024-03-04T16:27:26.511729+0000","last_clean":"2024-03-04T16:27:26.511729+0000","last_became_active":"2024-03-04T16:25:59.520911+0000","last_became_peered":"2024-03-04T16:25:59.520911+0000","last_unstale":"2024-03-04T16:27:26.511729+0000","last_undegraded":"2024-03-04T16:27:26.511729+0000","last_fullsized":"2024-03-04T16:27:26.511729+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:03.801685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:03.801685+0000","last_clean_scrub_stamp":"2024-03-04T16:27:03.801685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.017193+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":997,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595933+0000","last_change":"2024-03-04T16:27:07.790924+0000","last_active":"2024-03-04T16:27:26.595933+0000","last_peered":"2024-03-04T16:27:26.595933+0000","last_clean":"2024-03-04T16:27:26.595933+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:27:26.595933+0000","last_undegraded":"2024-03-04T16:27:26.595933+0000","last_fullsized":"2024-03-04T16:27:26.595933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.790846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:40.696222+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.790846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.927480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":869,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596682+0000","last_change":"2024-03-04T16:26:57.875935+0000","last_active":"2024-03-04T16:27:26.596682+0000","last_peered":"2024-03-04T16:27:26.596682+0000","last_clean":"2024-03-04T16:27:26.596682+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:27:26.596682+0000","last_undegraded":"2024-03-04T16:27:26.596682+0000","last_fullsized":"2024-03-04T16:27:26.596682+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:57.875855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:13.597504+0000","last_clean_scrub_stamp":"2024-03-04T16:26:57.875855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:18.783757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511083+0000","last_change":"2024-03-04T16:26:54.693305+0000","last_active":"2024-03-04T16:27:26.511083+0000","last_peered":"2024-03-04T16:27:26.511083+0000","last_clean":"2024-03-04T16:27:26.511083+0000","last_became_active":"2024-03-04T16:25:41.956367+0000","last_became_peered":"2024-03-04T16:25:41.956367+0000","last_unstale":"2024-03-04T16:27:26.511083+0000","last_undegraded":"2024-03-04T16:27:26.511083+0000","last_fullsized":"2024-03-04T16:27:26.511083+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:54.693176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.693176+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:15.427563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":88,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966428+0000","last_change":"2024-03-04T16:27:10.127287+0000","last_active":"2024-03-04T16:27:24.966428+0000","last_peered":"2024-03-04T16:27:24.966428+0000","last_clean":"2024-03-04T16:27:24.966428+0000","last_became_active":"2024-03-04T16:25:35.906317+0000","last_became_peered":"2024-03-04T16:25:35.906317+0000","last_unstale":"2024-03-04T16:27:24.966428+0000","last_undegraded":"2024-03-04T16:27:24.966428+0000","last_fullsized":"2024-03-04T16:27:24.966428+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:10.127173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.127173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.871370+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511166+0000","last_change":"2024-03-04T16:26:50.816244+0000","last_active":"2024-03-04T16:27:26.511166+0000","last_peered":"2024-03-04T16:27:26.511166+0000","last_clean":"2024-03-04T16:27:26.511166+0000","last_became_active":"2024-03-04T16:25:41.956943+0000","last_became_peered":"2024-03-04T16:25:41.956943+0000","last_unstale":"2024-03-04T16:27:26.511166+0000","last_undegraded":"2024-03-04T16:27:26.511166+0000","last_fullsized":"2024-03-04T16:27:26.511166+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:50.816139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:26:50.816139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:15.755176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":80,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966705+0000","last_change":"2024-03-04T16:26:26.735107+0000","last_active":"2024-03-04T16:27:24.966705+0000","last_peered":"2024-03-04T16:27:24.966705+0000","last_clean":"2024-03-04T16:27:24.966705+0000","last_became_active":"2024-03-04T16:25:35.901847+0000","last_became_peered":"2024-03-04T16:25:35.901847+0000","last_unstale":"2024-03-04T16:27:24.966705+0000","last_undegraded":"2024-03-04T16:27:24.966705+0000","last_fullsized":"2024-03-04T16:27:24.966705+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.735010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.735010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:37.980069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'39","reported_seq":995,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966075+0000","last_change":"2024-03-04T16:26:30.685418+0000","last_active":"2024-03-04T16:27:24.966075+0000","last_peered":"2024-03-04T16:27:24.966075+0000","last_clean":"2024-03-04T16:27:24.966075+0000","last_became_active":"2024-03-04T16:26:16.730442+0000","last_became_peered":"2024-03-04T16:26:16.730442+0000","last_unstale":"2024-03-04T16:27:24.966075+0000","last_undegraded":"2024-03-04T16:27:24.966075+0000","last_fullsized":"2024-03-04T16:27:24.966075+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:26:30.685312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.685312+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.648562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":850,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595872+0000","last_change":"2024-03-04T16:26:29.845038+0000","last_active":"2024-03-04T16:27:26.595872+0000","last_peered":"2024-03-04T16:27:26.595872+0000","last_clean":"2024-03-04T16:27:26.595872+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:27:26.595872+0000","last_undegraded":"2024-03-04T16:27:26.595872+0000","last_fullsized":"2024-03-04T16:27:26.595872+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:29.844928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:26:29.844928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:36.972512+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":989,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597041+0000","last_change":"2024-03-04T16:26:19.061141+0000","last_active":"2024-03-04T16:27:26.597041+0000","last_peered":"2024-03-04T16:27:26.597041+0000","last_clean":"2024-03-04T16:27:26.597041+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:27:26.597041+0000","last_undegraded":"2024-03-04T16:27:26.597041+0000","last_fullsized":"2024-03-04T16:27:26.597041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.061034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.061034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598188+0000","last_change":"2024-03-04T16:27:20.063727+0000","last_active":"2024-03-04T16:27:26.598188+0000","last_peered":"2024-03-04T16:27:26.598188+0000","last_clean":"2024-03-04T16:27:26.598188+0000","last_became_active":"2024-03-04T16:25:41.969493+0000","last_became_peered":"2024-03-04T16:25:41.969493+0000","last_unstale":"2024-03-04T16:27:26.598188+0000","last_undegraded":"2024-03-04T16:27:26.598188+0000","last_fullsized":"2024-03-04T16:27:26.598188+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:20.063653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:20.063653+0000","last_clean_scrub_stamp":"2024-03-04T16:27:20.063653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:40.102745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'25","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511734+0000","last_change":"2024-03-04T16:27:08.824922+0000","last_active":"2024-03-04T16:27:26.511734+0000","last_peered":"2024-03-04T16:27:26.511734+0000","last_clean":"2024-03-04T16:27:26.511734+0000","last_became_active":"2024-03-04T16:25:41.956644+0000","last_became_peered":"2024-03-04T16:25:41.956644+0000","last_unstale":"2024-03-04T16:27:26.511734+0000","last_undegraded":"2024-03-04T16:27:26.511734+0000","last_fullsized":"2024-03-04T16:27:26.511734+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:27:08.824811+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.824811+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.204239+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596426+0000","last_change":"2024-03-04T16:27:21.132837+0000","last_active":"2024-03-04T16:27:26.596426+0000","last_peered":"2024-03-04T16:27:26.596426+0000","last_clean":"2024-03-04T16:27:26.596426+0000","last_became_active":"2024-03-04T16:26:25.882740+0000","last_became_peered":"2024-03-04T16:26:25.882740+0000","last_unstale":"2024-03-04T16:27:26.596426+0000","last_undegraded":"2024-03-04T16:27:26.596426+0000","last_fullsized":"2024-03-04T16:27:26.596426+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:21.132730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.132730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.971036+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1018,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597061+0000","last_change":"2024-03-04T16:26:56.982294+0000","last_active":"2024-03-04T16:27:26.597061+0000","last_peered":"2024-03-04T16:27:26.597061+0000","last_clean":"2024-03-04T16:27:26.597061+0000","last_became_active":"2024-03-04T16:25:41.959358+0000","last_became_peered":"2024-03-04T16:25:41.959358+0000","last_unstale":"2024-03-04T16:27:26.597061+0000","last_undegraded":"2024-03-04T16:27:26.597061+0000","last_fullsized":"2024-03-04T16:27:26.597061+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:56.982182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:39.966717+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.982182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.940084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1029,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597397+0000","last_change":"2024-03-04T16:27:14.138515+0000","last_active":"2024-03-04T16:27:26.597397+0000","last_peered":"2024-03-04T16:27:26.597397+0000","last_clean":"2024-03-04T16:27:26.597397+0000","last_became_active":"2024-03-04T16:25:41.969615+0000","last_became_peered":"2024-03-04T16:25:41.969615+0000","last_unstale":"2024-03-04T16:27:26.597397+0000","last_undegraded":"2024-03-04T16:27:26.597397+0000","last_fullsized":"2024-03-04T16:27:26.597397+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:14.138418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:14.138418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.285858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"441'11","reported_seq":104,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511917+0000","last_change":"2024-03-04T16:27:17.842174+0000","last_active":"2024-03-04T16:27:26.511917+0000","last_peered":"2024-03-04T16:27:26.511917+0000","last_clean":"2024-03-04T16:27:26.511917+0000","last_became_active":"2024-03-04T16:25:59.529024+0000","last_became_peered":"2024-03-04T16:25:59.529024+0000","last_unstale":"2024-03-04T16:27:26.511917+0000","last_undegraded":"2024-03-04T16:27:26.511917+0000","last_fullsized":"2024-03-04T16:27:26.511917+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'11","last_scrub_stamp":"2024-03-04T16:27:17.842067+0000","last_deep_scrub":"441'11","last_deep_scrub_stamp":"2024-03-04T16:26:10.685697+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.842067+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:22.385108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512167+0000","last_change":"2024-03-04T16:26:51.856186+0000","last_active":"2024-03-04T16:27:26.512167+0000","last_peered":"2024-03-04T16:27:26.512167+0000","last_clean":"2024-03-04T16:27:26.512167+0000","last_became_active":"2024-03-04T16:25:41.963647+0000","last_became_peered":"2024-03-04T16:25:41.963647+0000","last_unstale":"2024-03-04T16:27:26.512167+0000","last_undegraded":"2024-03-04T16:27:26.512167+0000","last_fullsized":"2024-03-04T16:27:26.512167+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:51.856082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:51.856082+0000","last_clean_scrub_stamp":"2024-03-04T16:26:51.856082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:13.230502+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597041+0000","last_change":"2024-03-04T16:26:55.771511+0000","last_active":"2024-03-04T16:27:26.597041+0000","last_peered":"2024-03-04T16:27:26.597041+0000","last_clean":"2024-03-04T16:27:26.597041+0000","last_became_active":"2024-03-04T16:25:41.958642+0000","last_became_peered":"2024-03-04T16:25:41.958642+0000","last_unstale":"2024-03-04T16:27:26.597041+0000","last_undegraded":"2024-03-04T16:27:26.597041+0000","last_fullsized":"2024-03-04T16:27:26.597041+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:55.771436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.771436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:16.232921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":873,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596336+0000","last_change":"2024-03-04T16:27:25.879064+0000","last_active":"2024-03-04T16:27:26.596336+0000","last_peered":"2024-03-04T16:27:26.596336+0000","last_clean":"2024-03-04T16:27:26.596336+0000","last_became_active":"2024-03-04T16:25:41.955200+0000","last_became_peered":"2024-03-04T16:25:41.955200+0000","last_unstale":"2024-03-04T16:27:26.596336+0000","last_undegraded":"2024-03-04T16:27:26.596336+0000","last_fullsized":"2024-03-04T16:27:26.596336+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:25.878983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:25.878983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:40.924025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":490,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510527+0000","last_change":"2024-03-04T16:26:47.760763+0000","last_active":"2024-03-04T16:27:26.510527+0000","last_peered":"2024-03-04T16:27:26.510527+0000","last_clean":"2024-03-04T16:27:26.510527+0000","last_became_active":"2024-03-04T16:25:35.903214+0000","last_became_peered":"2024-03-04T16:25:35.903214+0000","last_unstale":"2024-03-04T16:27:26.510527+0000","last_undegraded":"2024-03-04T16:27:26.510527+0000","last_fullsized":"2024-03-04T16:27:26.510527+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:26:47.760650+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:26:47.760650+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.335879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":272,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.967077+0000","last_change":"2024-03-04T16:26:25.887125+0000","last_active":"2024-03-04T16:27:24.967077+0000","last_peered":"2024-03-04T16:27:24.967077+0000","last_clean":"2024-03-04T16:27:24.967077+0000","last_became_active":"2024-03-04T16:26:25.884116+0000","last_became_peered":"2024-03-04T16:26:25.884116+0000","last_unstale":"2024-03-04T16:27:24.967077+0000","last_undegraded":"2024-03-04T16:27:24.967077+0000","last_fullsized":"2024-03-04T16:27:24.967077+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.974587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.974587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511843+0000","last_change":"2024-03-04T16:26:31.692570+0000","last_active":"2024-03-04T16:27:26.511843+0000","last_peered":"2024-03-04T16:27:26.511843+0000","last_clean":"2024-03-04T16:27:26.511843+0000","last_became_active":"2024-03-04T16:25:41.966433+0000","last_became_peered":"2024-03-04T16:25:41.966433+0000","last_unstale":"2024-03-04T16:27:26.511843+0000","last_undegraded":"2024-03-04T16:27:26.511843+0000","last_fullsized":"2024-03-04T16:27:26.511843+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:31.692472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.692472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:37.869074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511373+0000","last_change":"2024-03-04T16:26:22.749558+0000","last_active":"2024-03-04T16:27:26.511373+0000","last_peered":"2024-03-04T16:27:26.511373+0000","last_clean":"2024-03-04T16:27:26.511373+0000","last_became_active":"2024-03-04T16:25:41.965271+0000","last_became_peered":"2024-03-04T16:25:41.965271+0000","last_unstale":"2024-03-04T16:27:26.511373+0000","last_undegraded":"2024-03-04T16:27:26.511373+0000","last_fullsized":"2024-03-04T16:27:26.511373+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:22.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:22.749468+0000","last_clean_scrub_stamp":"2024-03-04T16:26:22.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:49.052669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":728,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966065+0000","last_change":"2024-03-04T16:26:54.787704+0000","last_active":"2024-03-04T16:27:24.966065+0000","last_peered":"2024-03-04T16:27:24.966065+0000","last_clean":"2024-03-04T16:27:24.966065+0000","last_became_active":"2024-03-04T16:26:16.728742+0000","last_became_peered":"2024-03-04T16:26:16.728742+0000","last_unstale":"2024-03-04T16:27:24.966065+0000","last_undegraded":"2024-03-04T16:27:24.966065+0000","last_fullsized":"2024-03-04T16:27:24.966065+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:26:54.787570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.787570+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.943006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":1016,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597863+0000","last_change":"2024-03-04T16:26:43.030101+0000","last_active":"2024-03-04T16:27:26.597863+0000","last_peered":"2024-03-04T16:27:26.597863+0000","last_clean":"2024-03-04T16:27:26.597863+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:27:26.597863+0000","last_undegraded":"2024-03-04T16:27:26.597863+0000","last_fullsized":"2024-03-04T16:27:26.597863+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:26:43.029998+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:25:13.109799+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.029998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.200254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"491'14","reported_seq":343,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598461+0000","last_change":"2024-03-04T16:26:56.789600+0000","last_active":"2024-03-04T16:27:26.598461+0000","last_peered":"2024-03-04T16:27:26.598461+0000","last_clean":"2024-03-04T16:27:26.598461+0000","last_became_active":"2024-03-04T16:25:41.958850+0000","last_became_peered":"2024-03-04T16:25:41.958850+0000","last_unstale":"2024-03-04T16:27:26.598461+0000","last_undegraded":"2024-03-04T16:27:26.598461+0000","last_fullsized":"2024-03-04T16:27:26.598461+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'14","last_scrub_stamp":"2024-03-04T16:26:56.789492+0000","last_deep_scrub":"491'14","last_deep_scrub_stamp":"2024-03-04T16:26:56.789492+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.789492+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:09.530922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1064,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596193+0000","last_change":"2024-03-04T16:27:06.794156+0000","last_active":"2024-03-04T16:27:26.596193+0000","last_peered":"2024-03-04T16:27:26.596193+0000","last_clean":"2024-03-04T16:27:26.596193+0000","last_became_active":"2024-03-04T16:25:41.957831+0000","last_became_peered":"2024-03-04T16:25:41.957831+0000","last_unstale":"2024-03-04T16:27:26.596193+0000","last_undegraded":"2024-03-04T16:27:26.596193+0000","last_fullsized":"2024-03-04T16:27:26.596193+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:27:06.794074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.794074+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.275696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":929,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596904+0000","last_change":"2024-03-04T16:26:24.745956+0000","last_active":"2024-03-04T16:27:26.596904+0000","last_peered":"2024-03-04T16:27:26.596904+0000","last_clean":"2024-03-04T16:27:26.596904+0000","last_became_active":"2024-03-04T16:25:35.916024+0000","last_became_peered":"2024-03-04T16:25:35.916024+0000","last_unstale":"2024-03-04T16:27:26.596904+0000","last_undegraded":"2024-03-04T16:27:26.596904+0000","last_fullsized":"2024-03-04T16:27:26.596904+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:26:24.745848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:24.745848+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:44.037802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1004,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597110+0000","last_change":"2024-03-04T16:26:50.064603+0000","last_active":"2024-03-04T16:27:26.597110+0000","last_peered":"2024-03-04T16:27:26.597110+0000","last_clean":"2024-03-04T16:27:26.597110+0000","last_became_active":"2024-03-04T16:25:41.957834+0000","last_became_peered":"2024-03-04T16:25:41.957834+0000","last_unstale":"2024-03-04T16:27:26.597110+0000","last_undegraded":"2024-03-04T16:27:26.597110+0000","last_fullsized":"2024-03-04T16:27:26.597110+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:50.064451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:08.786009+0000","last_clean_scrub_stamp":"2024-03-04T16:26:50.064451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:51.834581+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":527,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596433+0000","last_change":"2024-03-04T16:26:17.697728+0000","last_active":"2024-03-04T16:27:26.596433+0000","last_peered":"2024-03-04T16:27:26.596433+0000","last_clean":"2024-03-04T16:27:26.596433+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:27:26.596433+0000","last_undegraded":"2024-03-04T16:27:26.596433+0000","last_fullsized":"2024-03-04T16:27:26.596433+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.697632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:17.697632+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.697632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.746709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":86,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.965125+0000","last_change":"2024-03-04T16:26:40.708684+0000","last_active":"2024-03-04T16:27:24.965125+0000","last_peered":"2024-03-04T16:27:24.965125+0000","last_clean":"2024-03-04T16:27:24.965125+0000","last_became_active":"2024-03-04T16:25:59.530296+0000","last_became_peered":"2024-03-04T16:25:59.530296+0000","last_unstale":"2024-03-04T16:27:24.965125+0000","last_undegraded":"2024-03-04T16:27:24.965125+0000","last_fullsized":"2024-03-04T16:27:24.965125+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.708597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.708597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.588913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'27","reported_seq":293,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.747890+0000","last_change":"2024-03-04T16:27:26.747928+0000","last_active":"2024-03-04T16:27:26.747890+0000","last_peered":"2024-03-04T16:27:26.747890+0000","last_clean":"2024-03-04T16:27:26.747890+0000","last_became_active":"2024-03-04T16:21:45.434840+0000","last_became_peered":"2024-03-04T16:21:45.434840+0000","last_unstale":"2024-03-04T16:27:26.747890+0000","last_undegraded":"2024-03-04T16:27:26.747890+0000","last_fullsized":"2024-03-04T16:27:26.747890+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:27:26.747811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:26.747811+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.588462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1045,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597193+0000","last_change":"2024-03-04T16:26:40.060557+0000","last_active":"2024-03-04T16:27:26.597193+0000","last_peered":"2024-03-04T16:27:26.597193+0000","last_clean":"2024-03-04T16:27:26.597193+0000","last_became_active":"2024-03-04T16:25:41.959916+0000","last_became_peered":"2024-03-04T16:25:41.959916+0000","last_unstale":"2024-03-04T16:27:26.597193+0000","last_undegraded":"2024-03-04T16:27:26.597193+0000","last_fullsized":"2024-03-04T16:27:26.597193+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.060432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.060432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.616665+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596639+0000","last_change":"2024-03-04T16:26:47.800409+0000","last_active":"2024-03-04T16:27:26.596639+0000","last_peered":"2024-03-04T16:27:26.596639+0000","last_clean":"2024-03-04T16:27:26.596639+0000","last_became_active":"2024-03-04T16:25:41.958854+0000","last_became_peered":"2024-03-04T16:25:41.958854+0000","last_unstale":"2024-03-04T16:27:26.596639+0000","last_undegraded":"2024-03-04T16:27:26.596639+0000","last_fullsized":"2024-03-04T16:27:26.596639+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:47.800303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:26:47.800303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.203816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":862,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596590+0000","last_change":"2024-03-04T16:26:23.724658+0000","last_active":"2024-03-04T16:27:26.596590+0000","last_peered":"2024-03-04T16:27:26.596590+0000","last_clean":"2024-03-04T16:27:26.596590+0000","last_became_active":"2024-03-04T16:25:35.903977+0000","last_became_peered":"2024-03-04T16:25:35.903977+0000","last_unstale":"2024-03-04T16:27:26.596590+0000","last_undegraded":"2024-03-04T16:27:26.596590+0000","last_fullsized":"2024-03-04T16:27:26.596590+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:23.724559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:11.629031+0000","last_clean_scrub_stamp":"2024-03-04T16:26:23.724559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:40.852625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"458'14","reported_seq":94,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.964812+0000","last_change":"2024-03-04T16:27:10.900226+0000","last_active":"2024-03-04T16:27:24.964812+0000","last_peered":"2024-03-04T16:27:24.964812+0000","last_clean":"2024-03-04T16:27:24.964812+0000","last_became_active":"2024-03-04T16:25:35.913075+0000","last_became_peered":"2024-03-04T16:25:35.913075+0000","last_unstale":"2024-03-04T16:27:24.964812+0000","last_undegraded":"2024-03-04T16:27:24.964812+0000","last_fullsized":"2024-03-04T16:27:24.964812+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'14","last_scrub_stamp":"2024-03-04T16:27:10.900101+0000","last_deep_scrub":"458'14","last_deep_scrub_stamp":"2024-03-04T16:24:16.155381+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.900101+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:36.184135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":80,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966669+0000","last_change":"2024-03-04T16:26:28.901269+0000","last_active":"2024-03-04T16:27:24.966669+0000","last_peered":"2024-03-04T16:27:24.966669+0000","last_clean":"2024-03-04T16:27:24.966669+0000","last_became_active":"2024-03-04T16:25:35.912277+0000","last_became_peered":"2024-03-04T16:25:35.912277+0000","last_unstale":"2024-03-04T16:27:24.966669+0000","last_undegraded":"2024-03-04T16:27:24.966669+0000","last_fullsized":"2024-03-04T16:27:24.966669+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:28.901143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:26:28.901143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:32.682436+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1023,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596959+0000","last_change":"2024-03-04T16:26:25.855926+0000","last_active":"2024-03-04T16:27:26.596959+0000","last_peered":"2024-03-04T16:27:26.596959+0000","last_clean":"2024-03-04T16:27:26.596959+0000","last_became_active":"2024-03-04T16:25:41.951860+0000","last_became_peered":"2024-03-04T16:25:41.951860+0000","last_unstale":"2024-03-04T16:27:26.596959+0000","last_undegraded":"2024-03-04T16:27:26.596959+0000","last_fullsized":"2024-03-04T16:27:26.596959+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:25.855795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:19.018504+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.855795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.161793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1024,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598630+0000","last_change":"2024-03-04T16:26:16.984149+0000","last_active":"2024-03-04T16:27:26.598630+0000","last_peered":"2024-03-04T16:27:26.598630+0000","last_clean":"2024-03-04T16:27:26.598630+0000","last_became_active":"2024-03-04T16:25:59.527423+0000","last_became_peered":"2024-03-04T16:25:59.527423+0000","last_unstale":"2024-03-04T16:27:26.598630+0000","last_undegraded":"2024-03-04T16:27:26.598630+0000","last_fullsized":"2024-03-04T16:27:26.598630+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:16.984069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:16.984069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:33.356645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'27","reported_seq":100,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.965905+0000","last_change":"2024-03-04T16:27:19.894464+0000","last_active":"2024-03-04T16:27:24.965905+0000","last_peered":"2024-03-04T16:27:24.965905+0000","last_clean":"2024-03-04T16:27:24.965905+0000","last_became_active":"2024-03-04T16:25:35.909109+0000","last_became_peered":"2024-03-04T16:25:35.909109+0000","last_unstale":"2024-03-04T16:27:24.965905+0000","last_undegraded":"2024-03-04T16:27:24.965905+0000","last_fullsized":"2024-03-04T16:27:24.965905+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:27:19.894371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:19.894371+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:37.769683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":862,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:28.908219+0000","last_change":"2024-03-04T16:27:28.908243+0000","last_active":"2024-03-04T16:27:28.908219+0000","last_peered":"2024-03-04T16:27:28.908219+0000","last_clean":"2024-03-04T16:27:28.908219+0000","last_became_active":"2024-03-04T16:26:25.886675+0000","last_became_peered":"2024-03-04T16:26:25.886675+0000","last_unstale":"2024-03-04T16:27:28.908219+0000","last_undegraded":"2024-03-04T16:27:28.908219+0000","last_fullsized":"2024-03-04T16:27:28.908219+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:28.908136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:12.785337+0000","last_clean_scrub_stamp":"2024-03-04T16:27:28.908136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:45.088043+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":592,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966247+0000","last_change":"2024-03-04T16:26:38.680384+0000","last_active":"2024-03-04T16:27:24.966247+0000","last_peered":"2024-03-04T16:27:24.966247+0000","last_clean":"2024-03-04T16:27:24.966247+0000","last_became_active":"2024-03-04T16:26:16.732699+0000","last_became_peered":"2024-03-04T16:26:16.732699+0000","last_unstale":"2024-03-04T16:27:24.966247+0000","last_undegraded":"2024-03-04T16:27:24.966247+0000","last_fullsized":"2024-03-04T16:27:24.966247+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:38.680307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:38.680307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:48.850265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596843+0000","last_change":"2024-03-04T16:26:35.972842+0000","last_active":"2024-03-04T16:27:26.596843+0000","last_peered":"2024-03-04T16:27:26.596843+0000","last_clean":"2024-03-04T16:27:26.596843+0000","last_became_active":"2024-03-04T16:25:41.950167+0000","last_became_peered":"2024-03-04T16:25:41.950167+0000","last_unstale":"2024-03-04T16:27:26.596843+0000","last_undegraded":"2024-03-04T16:27:26.596843+0000","last_fullsized":"2024-03-04T16:27:26.596843+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:35.972703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:35.972703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:47.372306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'25","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596157+0000","last_change":"2024-03-04T16:26:53.747035+0000","last_active":"2024-03-04T16:27:26.596157+0000","last_peered":"2024-03-04T16:27:26.596157+0000","last_clean":"2024-03-04T16:27:26.596157+0000","last_became_active":"2024-03-04T16:25:59.531329+0000","last_became_peered":"2024-03-04T16:25:59.531329+0000","last_unstale":"2024-03-04T16:27:26.596157+0000","last_undegraded":"2024-03-04T16:27:26.596157+0000","last_fullsized":"2024-03-04T16:27:26.596157+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:26:53.746921+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:26:53.746921+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.609935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":412,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.965796+0000","last_change":"2024-03-04T16:27:13.954062+0000","last_active":"2024-03-04T16:27:24.965796+0000","last_peered":"2024-03-04T16:27:24.965796+0000","last_clean":"2024-03-04T16:27:24.965796+0000","last_became_active":"2024-03-04T16:26:16.731876+0000","last_became_peered":"2024-03-04T16:26:16.731876+0000","last_unstale":"2024-03-04T16:27:24.965796+0000","last_undegraded":"2024-03-04T16:27:24.965796+0000","last_fullsized":"2024-03-04T16:27:24.965796+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:13.953944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:13.953944+0000","last_clean_scrub_stamp":"2024-03-04T16:27:13.953944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.612571+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1009,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596611+0000","last_change":"2024-03-04T16:26:11.825312+0000","last_active":"2024-03-04T16:27:26.596611+0000","last_peered":"2024-03-04T16:27:26.596611+0000","last_clean":"2024-03-04T16:27:26.596611+0000","last_became_active":"2024-03-04T16:25:35.914323+0000","last_became_peered":"2024-03-04T16:25:35.914323+0000","last_unstale":"2024-03-04T16:27:26.596611+0000","last_undegraded":"2024-03-04T16:27:26.596611+0000","last_fullsized":"2024-03-04T16:27:26.596611+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:11.825229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:11.825229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:30.849743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511698+0000","last_change":"2024-03-04T16:26:19.810601+0000","last_active":"2024-03-04T16:27:26.511698+0000","last_peered":"2024-03-04T16:27:26.511698+0000","last_clean":"2024-03-04T16:27:26.511698+0000","last_became_active":"2024-03-04T16:25:41.960053+0000","last_became_peered":"2024-03-04T16:25:41.960053+0000","last_unstale":"2024-03-04T16:27:26.511698+0000","last_undegraded":"2024-03-04T16:27:26.511698+0000","last_fullsized":"2024-03-04T16:27:26.511698+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.810508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.810508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:46.245891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511753+0000","last_change":"2024-03-04T16:26:21.733482+0000","last_active":"2024-03-04T16:27:26.511753+0000","last_peered":"2024-03-04T16:27:26.511753+0000","last_clean":"2024-03-04T16:27:26.511753+0000","last_became_active":"2024-03-04T16:25:41.958985+0000","last_became_peered":"2024-03-04T16:25:41.958985+0000","last_unstale":"2024-03-04T16:27:26.511753+0000","last_undegraded":"2024-03-04T16:27:26.511753+0000","last_fullsized":"2024-03-04T16:27:26.511753+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:26:21.733338+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-04T16:26:21.733338+0000","last_clean_scrub_stamp":"2024-03-04T16:26:21.733338+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:40.813524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":871,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596526+0000","last_change":"2024-03-04T16:27:06.000477+0000","last_active":"2024-03-04T16:27:26.596526+0000","last_peered":"2024-03-04T16:27:26.596526+0000","last_clean":"2024-03-04T16:27:26.596526+0000","last_became_active":"2024-03-04T16:25:56.057550+0000","last_became_peered":"2024-03-04T16:25:56.057550+0000","last_unstale":"2024-03-04T16:27:26.596526+0000","last_undegraded":"2024-03-04T16:27:26.596526+0000","last_fullsized":"2024-03-04T16:27:26.596526+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:06.000373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:06.000373+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.000373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:09.656090+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"484'11","reported_seq":1328,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598304+0000","last_change":"2024-03-04T16:26:18.033443+0000","last_active":"2024-03-04T16:27:26.598304+0000","last_peered":"2024-03-04T16:27:26.598304+0000","last_clean":"2024-03-04T16:27:26.598304+0000","last_became_active":"2024-03-04T16:25:25.762427+0000","last_became_peered":"2024-03-04T16:25:25.762427+0000","last_unstale":"2024-03-04T16:27:26.598304+0000","last_undegraded":"2024-03-04T16:27:26.598304+0000","last_fullsized":"2024-03-04T16:27:26.598304+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'11","last_scrub_stamp":"2024-03-04T16:26:18.033357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:18.033357+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.480149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"517'2","reported_seq":1082,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597956+0000","last_change":"2024-03-04T16:27:04.851938+0000","last_active":"2024-03-04T16:27:26.597956+0000","last_peered":"2024-03-04T16:27:26.597956+0000","last_clean":"2024-03-04T16:27:26.597956+0000","last_became_active":"2024-03-04T16:25:41.951420+0000","last_became_peered":"2024-03-04T16:25:41.951420+0000","last_unstale":"2024-03-04T16:27:26.597956+0000","last_undegraded":"2024-03-04T16:27:26.597956+0000","last_fullsized":"2024-03-04T16:27:26.597956+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'2","last_scrub_stamp":"2024-03-04T16:27:04.851837+0000","last_deep_scrub":"517'2","last_deep_scrub_stamp":"2024-03-04T16:27:04.851837+0000","last_clean_scrub_stamp":"2024-03-04T16:27:04.851837+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.303458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'30","reported_seq":635,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966902+0000","last_change":"2024-03-04T16:27:21.893403+0000","last_active":"2024-03-04T16:27:24.966902+0000","last_peered":"2024-03-04T16:27:24.966902+0000","last_clean":"2024-03-04T16:27:24.966902+0000","last_became_active":"2024-03-04T16:26:16.732368+0000","last_became_peered":"2024-03-04T16:26:16.732368+0000","last_unstale":"2024-03-04T16:27:24.966902+0000","last_undegraded":"2024-03-04T16:27:24.966902+0000","last_fullsized":"2024-03-04T16:27:24.966902+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:27:21.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.893298+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.899355+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":87,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512416+0000","last_change":"2024-03-04T16:27:20.014282+0000","last_active":"2024-03-04T16:27:26.512416+0000","last_peered":"2024-03-04T16:27:26.512416+0000","last_clean":"2024-03-04T16:27:26.512416+0000","last_became_active":"2024-03-04T16:25:41.965635+0000","last_became_peered":"2024-03-04T16:25:41.965635+0000","last_unstale":"2024-03-04T16:27:26.512416+0000","last_undegraded":"2024-03-04T16:27:26.512416+0000","last_fullsized":"2024-03-04T16:27:26.512416+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:20.014158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:27:20.014158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:31.341699+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"489'13","reported_seq":106,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.967330+0000","last_change":"2024-03-04T16:26:36.670294+0000","last_active":"2024-03-04T16:27:24.967330+0000","last_peered":"2024-03-04T16:27:24.967330+0000","last_clean":"2024-03-04T16:27:24.967330+0000","last_became_active":"2024-03-04T16:25:35.905277+0000","last_became_peered":"2024-03-04T16:25:35.905277+0000","last_unstale":"2024-03-04T16:27:24.967330+0000","last_undegraded":"2024-03-04T16:27:24.967330+0000","last_fullsized":"2024-03-04T16:27:24.967330+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'13","last_scrub_stamp":"2024-03-04T16:26:36.670181+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:26:36.670181+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.818094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"505'2","reported_seq":121,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596309+0000","last_change":"2024-03-04T16:26:15.743215+0000","last_active":"2024-03-04T16:27:26.596309+0000","last_peered":"2024-03-04T16:27:26.596309+0000","last_clean":"2024-03-04T16:27:26.596309+0000","last_became_active":"2024-03-04T16:25:56.057518+0000","last_became_peered":"2024-03-04T16:25:56.057518+0000","last_unstale":"2024-03-04T16:27:26.596309+0000","last_undegraded":"2024-03-04T16:27:26.596309+0000","last_fullsized":"2024-03-04T16:27:26.596309+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'2","last_scrub_stamp":"2024-03-04T16:26:15.743125+0000","last_deep_scrub":"505'2","last_deep_scrub_stamp":"2024-03-04T16:26:15.743125+0000","last_clean_scrub_stamp":"2024-03-04T16:26:15.743125+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.073501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"441'21","reported_seq":51,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597889+0000","last_change":"2024-03-04T16:26:58.913422+0000","last_active":"2024-03-04T16:27:26.597889+0000","last_peered":"2024-03-04T16:27:26.597889+0000","last_clean":"2024-03-04T16:27:26.597889+0000","last_became_active":"2024-03-04T16:26:25.882162+0000","last_became_peered":"2024-03-04T16:26:25.882162+0000","last_unstale":"2024-03-04T16:27:26.597889+0000","last_undegraded":"2024-03-04T16:27:26.597889+0000","last_fullsized":"2024-03-04T16:27:26.597889+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:58.913306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:58.913306+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:25.220282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":424,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510636+0000","last_change":"2024-03-04T16:26:27.578848+0000","last_active":"2024-03-04T16:27:26.510636+0000","last_peered":"2024-03-04T16:27:26.510636+0000","last_clean":"2024-03-04T16:27:26.510636+0000","last_became_active":"2024-03-04T16:25:41.957995+0000","last_became_peered":"2024-03-04T16:25:41.957995+0000","last_unstale":"2024-03-04T16:27:26.510636+0000","last_undegraded":"2024-03-04T16:27:26.510636+0000","last_fullsized":"2024-03-04T16:27:26.510636+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:27.578727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:27.578727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.415089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":80,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.967234+0000","last_change":"2024-03-04T16:26:13.739831+0000","last_active":"2024-03-04T16:27:24.967234+0000","last_peered":"2024-03-04T16:27:24.967234+0000","last_clean":"2024-03-04T16:27:24.967234+0000","last_became_active":"2024-03-04T16:25:35.913274+0000","last_became_peered":"2024-03-04T16:25:35.913274+0000","last_unstale":"2024-03-04T16:27:24.967234+0000","last_undegraded":"2024-03-04T16:27:24.967234+0000","last_fullsized":"2024-03-04T16:27:24.967234+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:13.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:13.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596903+0000","last_change":"2024-03-04T16:27:15.882985+0000","last_active":"2024-03-04T16:27:26.596903+0000","last_peered":"2024-03-04T16:27:26.596903+0000","last_clean":"2024-03-04T16:27:26.596903+0000","last_became_active":"2024-03-04T16:25:41.950697+0000","last_became_peered":"2024-03-04T16:25:41.950697+0000","last_unstale":"2024-03-04T16:27:26.596903+0000","last_undegraded":"2024-03-04T16:27:26.596903+0000","last_fullsized":"2024-03-04T16:27:26.596903+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:15.882861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:59.857747+0000","last_clean_scrub_stamp":"2024-03-04T16:27:15.882861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:29.402354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'21","reported_seq":910,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597010+0000","last_change":"2024-03-04T16:26:48.870663+0000","last_active":"2024-03-04T16:27:26.597010+0000","last_peered":"2024-03-04T16:27:26.597010+0000","last_clean":"2024-03-04T16:27:26.597010+0000","last_became_active":"2024-03-04T16:25:35.907039+0000","last_became_peered":"2024-03-04T16:25:35.907039+0000","last_unstale":"2024-03-04T16:27:26.597010+0000","last_undegraded":"2024-03-04T16:27:26.597010+0000","last_fullsized":"2024-03-04T16:27:26.597010+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:48.870514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.870514+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.628672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":105,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596855+0000","last_change":"2024-03-04T16:27:05.929412+0000","last_active":"2024-03-04T16:27:26.596855+0000","last_peered":"2024-03-04T16:27:26.596855+0000","last_clean":"2024-03-04T16:27:26.596855+0000","last_became_active":"2024-03-04T16:25:41.958001+0000","last_became_peered":"2024-03-04T16:25:41.958001+0000","last_unstale":"2024-03-04T16:27:26.596855+0000","last_undegraded":"2024-03-04T16:27:26.596855+0000","last_fullsized":"2024-03-04T16:27:26.596855+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:05.929284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:27:05.929284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.779254+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598513+0000","last_change":"2024-03-04T16:27:09.067065+0000","last_active":"2024-03-04T16:27:26.598513+0000","last_peered":"2024-03-04T16:27:26.598513+0000","last_clean":"2024-03-04T16:27:26.598513+0000","last_became_active":"2024-03-04T16:25:35.915465+0000","last_became_peered":"2024-03-04T16:25:35.915465+0000","last_unstale":"2024-03-04T16:27:26.598513+0000","last_undegraded":"2024-03-04T16:27:26.598513+0000","last_fullsized":"2024-03-04T16:27:26.598513+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:09.066955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:27:09.066955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:25.935375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":279,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596446+0000","last_change":"2024-03-04T16:27:00.672881+0000","last_active":"2024-03-04T16:27:26.596446+0000","last_peered":"2024-03-04T16:27:26.596446+0000","last_clean":"2024-03-04T16:27:26.596446+0000","last_became_active":"2024-03-04T16:26:25.884905+0000","last_became_peered":"2024-03-04T16:26:25.884905+0000","last_unstale":"2024-03-04T16:27:26.596446+0000","last_undegraded":"2024-03-04T16:27:26.596446+0000","last_fullsized":"2024-03-04T16:27:26.596446+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:00.672777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.672777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:05.739226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'21","reported_seq":96,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.967323+0000","last_change":"2024-03-04T16:27:00.813329+0000","last_active":"2024-03-04T16:27:24.967323+0000","last_peered":"2024-03-04T16:27:24.967323+0000","last_clean":"2024-03-04T16:27:24.967323+0000","last_became_active":"2024-03-04T16:25:35.912617+0000","last_became_peered":"2024-03-04T16:25:35.912617+0000","last_unstale":"2024-03-04T16:27:24.967323+0000","last_undegraded":"2024-03-04T16:27:24.967323+0000","last_fullsized":"2024-03-04T16:27:24.967323+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:27:00.813233+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.813233+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:26.664975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1013,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597243+0000","last_change":"2024-03-04T16:26:30.938999+0000","last_active":"2024-03-04T16:27:26.597243+0000","last_peered":"2024-03-04T16:27:26.597243+0000","last_clean":"2024-03-04T16:27:26.597243+0000","last_became_active":"2024-03-04T16:21:45.424279+0000","last_became_peered":"2024-03-04T16:21:45.424279+0000","last_unstale":"2024-03-04T16:27:26.597243+0000","last_undegraded":"2024-03-04T16:27:26.597243+0000","last_fullsized":"2024-03-04T16:27:26.597243+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:30.938891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.938891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.799044+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":1087,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597530+0000","last_change":"2024-03-04T16:27:17.895718+0000","last_active":"2024-03-04T16:27:26.597530+0000","last_peered":"2024-03-04T16:27:26.597530+0000","last_clean":"2024-03-04T16:27:26.597530+0000","last_became_active":"2024-03-04T16:25:41.951605+0000","last_became_peered":"2024-03-04T16:25:41.951605+0000","last_unstale":"2024-03-04T16:27:26.597530+0000","last_undegraded":"2024-03-04T16:27:26.597530+0000","last_fullsized":"2024-03-04T16:27:26.597530+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:27:17.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.895612+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.319967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":57,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596516+0000","last_change":"2024-03-04T16:27:10.069698+0000","last_active":"2024-03-04T16:27:26.596516+0000","last_peered":"2024-03-04T16:27:26.596516+0000","last_clean":"2024-03-04T16:27:26.596516+0000","last_became_active":"2024-03-04T16:26:25.877388+0000","last_became_peered":"2024-03-04T16:26:25.877388+0000","last_unstale":"2024-03-04T16:27:26.596516+0000","last_undegraded":"2024-03-04T16:27:26.596516+0000","last_fullsized":"2024-03-04T16:27:26.596516+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:27:10.069583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.069583+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.478046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"492'15","reported_seq":1374,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597223+0000","last_change":"2024-03-04T16:27:21.827455+0000","last_active":"2024-03-04T16:27:26.597223+0000","last_peered":"2024-03-04T16:27:26.597223+0000","last_clean":"2024-03-04T16:27:26.597223+0000","last_became_active":"2024-03-04T16:25:41.952470+0000","last_became_peered":"2024-03-04T16:25:41.952470+0000","last_unstale":"2024-03-04T16:27:26.597223+0000","last_undegraded":"2024-03-04T16:27:26.597223+0000","last_fullsized":"2024-03-04T16:27:26.597223+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'15","last_scrub_stamp":"2024-03-04T16:27:21.827308+0000","last_deep_scrub":"492'15","last_deep_scrub_stamp":"2024-03-04T16:25:52.919379+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.827308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.014368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":88,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966520+0000","last_change":"2024-03-04T16:27:09.184891+0000","last_active":"2024-03-04T16:27:24.966520+0000","last_peered":"2024-03-04T16:27:24.966520+0000","last_clean":"2024-03-04T16:27:24.966520+0000","last_became_active":"2024-03-04T16:25:35.912995+0000","last_became_peered":"2024-03-04T16:25:35.912995+0000","last_unstale":"2024-03-04T16:27:24.966520+0000","last_undegraded":"2024-03-04T16:27:24.966520+0000","last_fullsized":"2024-03-04T16:27:24.966520+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:09.184789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:52.766115+0000","last_clean_scrub_stamp":"2024-03-04T16:27:09.184789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.784865+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1027,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597834+0000","last_change":"2024-03-04T16:26:45.021081+0000","last_active":"2024-03-04T16:27:26.597834+0000","last_peered":"2024-03-04T16:27:26.597834+0000","last_clean":"2024-03-04T16:27:26.597834+0000","last_became_active":"2024-03-04T16:25:41.952002+0000","last_became_peered":"2024-03-04T16:25:41.952002+0000","last_unstale":"2024-03-04T16:27:26.597834+0000","last_undegraded":"2024-03-04T16:27:26.597834+0000","last_fullsized":"2024-03-04T16:27:26.597834+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:45.020970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:16.071086+0000","last_clean_scrub_stamp":"2024-03-04T16:26:45.020970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:02.566679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":116,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511346+0000","last_change":"2024-03-04T16:27:04.776193+0000","last_active":"2024-03-04T16:27:26.511346+0000","last_peered":"2024-03-04T16:27:26.511346+0000","last_clean":"2024-03-04T16:27:26.511346+0000","last_became_active":"2024-03-04T16:26:25.884201+0000","last_became_peered":"2024-03-04T16:26:25.884201+0000","last_unstale":"2024-03-04T16:27:26.511346+0000","last_undegraded":"2024-03-04T16:27:26.511346+0000","last_fullsized":"2024-03-04T16:27:26.511346+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:27:04.776063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:04.776063+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.513891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":418,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510095+0000","last_change":"2024-03-04T16:26:52.787335+0000","last_active":"2024-03-04T16:27:26.510095+0000","last_peered":"2024-03-04T16:27:26.510095+0000","last_clean":"2024-03-04T16:27:26.510095+0000","last_became_active":"2024-03-04T16:24:49.071183+0000","last_became_peered":"2024-03-04T16:24:49.071183+0000","last_unstale":"2024-03-04T16:27:26.510095+0000","last_undegraded":"2024-03-04T16:27:26.510095+0000","last_fullsized":"2024-03-04T16:27:26.510095+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:52.787244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.787244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:16.662941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597157+0000","last_change":"2024-03-04T16:26:17.877692+0000","last_active":"2024-03-04T16:27:26.597157+0000","last_peered":"2024-03-04T16:27:26.597157+0000","last_clean":"2024-03-04T16:27:26.597157+0000","last_became_active":"2024-03-04T16:25:35.873601+0000","last_became_peered":"2024-03-04T16:25:35.873601+0000","last_unstale":"2024-03-04T16:27:26.597157+0000","last_undegraded":"2024-03-04T16:27:26.597157+0000","last_fullsized":"2024-03-04T16:27:26.597157+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.877611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.877611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:28.351289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":424,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510361+0000","last_change":"2024-03-04T16:26:26.529455+0000","last_active":"2024-03-04T16:27:26.510361+0000","last_peered":"2024-03-04T16:27:26.510361+0000","last_clean":"2024-03-04T16:27:26.510361+0000","last_became_active":"2024-03-04T16:26:25.884184+0000","last_became_peered":"2024-03-04T16:26:25.884184+0000","last_unstale":"2024-03-04T16:27:26.510361+0000","last_undegraded":"2024-03-04T16:27:26.510361+0000","last_fullsized":"2024-03-04T16:27:26.510361+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.529374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:15.548884+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.529374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.510339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597466+0000","last_change":"2024-03-04T16:26:26.875537+0000","last_active":"2024-03-04T16:27:26.597466+0000","last_peered":"2024-03-04T16:27:26.597466+0000","last_clean":"2024-03-04T16:27:26.597466+0000","last_became_active":"2024-03-04T16:26:25.877942+0000","last_became_peered":"2024-03-04T16:26:25.877942+0000","last_unstale":"2024-03-04T16:27:26.597466+0000","last_undegraded":"2024-03-04T16:27:26.597466+0000","last_fullsized":"2024-03-04T16:27:26.597466+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.875453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:26.875453+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.875453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.978054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'52","reported_seq":339,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597674+0000","last_change":"2024-03-04T16:27:06.789992+0000","last_active":"2024-03-04T16:27:26.597674+0000","last_peered":"2024-03-04T16:27:26.597674+0000","last_clean":"2024-03-04T16:27:26.597674+0000","last_became_active":"2024-03-04T16:22:24.826414+0000","last_became_peered":"2024-03-04T16:22:24.826414+0000","last_unstale":"2024-03-04T16:27:26.597674+0000","last_undegraded":"2024-03-04T16:27:26.597674+0000","last_fullsized":"2024-03-04T16:27:26.597674+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'52","last_scrub_stamp":"2024-03-04T16:27:06.789852+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.789852+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:26.566175+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":559,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595963+0000","last_change":"2024-03-04T16:26:43.035072+0000","last_active":"2024-03-04T16:27:26.595963+0000","last_peered":"2024-03-04T16:27:26.595963+0000","last_clean":"2024-03-04T16:27:26.595963+0000","last_became_active":"2024-03-04T16:25:56.057201+0000","last_became_peered":"2024-03-04T16:25:56.057201+0000","last_unstale":"2024-03-04T16:27:26.595963+0000","last_undegraded":"2024-03-04T16:27:26.595963+0000","last_fullsized":"2024-03-04T16:27:26.595963+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:43.034972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:04.117139+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.034972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.501653+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":450,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510345+0000","last_change":"2024-03-04T16:27:18.888603+0000","last_active":"2024-03-04T16:27:26.510345+0000","last_peered":"2024-03-04T16:27:26.510345+0000","last_clean":"2024-03-04T16:27:26.510345+0000","last_became_active":"2024-03-04T16:25:41.957993+0000","last_became_peered":"2024-03-04T16:25:41.957993+0000","last_unstale":"2024-03-04T16:27:26.510345+0000","last_undegraded":"2024-03-04T16:27:26.510345+0000","last_fullsized":"2024-03-04T16:27:26.510345+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:18.888522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:27:18.888522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:30.252697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1030,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597499+0000","last_change":"2024-03-04T16:26:49.939876+0000","last_active":"2024-03-04T16:27:26.597499+0000","last_peered":"2024-03-04T16:27:26.597499+0000","last_clean":"2024-03-04T16:27:26.597499+0000","last_became_active":"2024-03-04T16:25:35.914551+0000","last_became_peered":"2024-03-04T16:25:35.914551+0000","last_unstale":"2024-03-04T16:27:26.597499+0000","last_undegraded":"2024-03-04T16:27:26.597499+0000","last_fullsized":"2024-03-04T16:27:26.597499+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:49.939785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:26:49.939785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:00.738998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"441'25","reported_seq":910,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596550+0000","last_change":"2024-03-04T16:26:04.596514+0000","last_active":"2024-03-04T16:27:26.596550+0000","last_peered":"2024-03-04T16:27:26.596550+0000","last_clean":"2024-03-04T16:27:26.596550+0000","last_became_active":"2024-03-04T16:21:10.564447+0000","last_became_peered":"2024-03-04T16:21:10.564447+0000","last_unstale":"2024-03-04T16:27:26.596550+0000","last_undegraded":"2024-03-04T16:27:26.596550+0000","last_fullsized":"2024-03-04T16:27:26.596550+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:26:04.596406+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:26:04.596406+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:32.761300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":86,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966619+0000","last_change":"2024-03-04T16:26:39.736912+0000","last_active":"2024-03-04T16:27:24.966619+0000","last_peered":"2024-03-04T16:27:24.966619+0000","last_clean":"2024-03-04T16:27:24.966619+0000","last_became_active":"2024-03-04T16:25:41.945618+0000","last_became_peered":"2024-03-04T16:25:41.945618+0000","last_unstale":"2024-03-04T16:27:24.966619+0000","last_undegraded":"2024-03-04T16:27:24.966619+0000","last_fullsized":"2024-03-04T16:27:24.966619+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:39.736804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:39.736804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.849666+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":1038,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596889+0000","last_change":"2024-03-04T16:27:18.099142+0000","last_active":"2024-03-04T16:27:26.596889+0000","last_peered":"2024-03-04T16:27:26.596889+0000","last_clean":"2024-03-04T16:27:26.596889+0000","last_became_active":"2024-03-04T16:21:45.425232+0000","last_became_peered":"2024-03-04T16:21:45.425232+0000","last_unstale":"2024-03-04T16:27:26.596889+0000","last_undegraded":"2024-03-04T16:27:26.596889+0000","last_fullsized":"2024-03-04T16:27:26.596889+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:27:18.099030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:27:18.099030+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:30.139552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511533+0000","last_change":"2024-03-04T16:26:43.800073+0000","last_active":"2024-03-04T16:27:26.511533+0000","last_peered":"2024-03-04T16:27:26.511533+0000","last_clean":"2024-03-04T16:27:26.511533+0000","last_became_active":"2024-03-04T16:25:41.959686+0000","last_became_peered":"2024-03-04T16:25:41.959686+0000","last_unstale":"2024-03-04T16:27:26.511533+0000","last_undegraded":"2024-03-04T16:27:26.511533+0000","last_fullsized":"2024-03-04T16:27:26.511533+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:43.799888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.799888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:44.665976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"496'59","reported_seq":109,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596458+0000","last_change":"2024-03-04T16:26:43.864588+0000","last_active":"2024-03-04T16:27:26.596458+0000","last_peered":"2024-03-04T16:27:26.596458+0000","last_clean":"2024-03-04T16:27:26.596458+0000","last_became_active":"2024-03-04T16:25:59.530455+0000","last_became_peered":"2024-03-04T16:25:59.530455+0000","last_unstale":"2024-03-04T16:27:26.596458+0000","last_undegraded":"2024-03-04T16:27:26.596458+0000","last_fullsized":"2024-03-04T16:27:26.596458+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'59","last_scrub_stamp":"2024-03-04T16:26:43.864484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.864484+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.976346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":869,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596743+0000","last_change":"2024-03-04T16:26:54.810925+0000","last_active":"2024-03-04T16:27:26.596743+0000","last_peered":"2024-03-04T16:27:26.596743+0000","last_clean":"2024-03-04T16:27:26.596743+0000","last_became_active":"2024-03-04T16:25:41.954560+0000","last_became_peered":"2024-03-04T16:25:41.954560+0000","last_unstale":"2024-03-04T16:27:26.596743+0000","last_undegraded":"2024-03-04T16:27:26.596743+0000","last_fullsized":"2024-03-04T16:27:26.596743+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:54.810827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.810827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:01.618062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597581+0000","last_change":"2024-03-04T16:26:19.981556+0000","last_active":"2024-03-04T16:27:26.597581+0000","last_peered":"2024-03-04T16:27:26.597581+0000","last_clean":"2024-03-04T16:27:26.597581+0000","last_became_active":"2024-03-04T16:25:35.905161+0000","last_became_peered":"2024-03-04T16:25:35.905161+0000","last_unstale":"2024-03-04T16:27:26.597581+0000","last_undegraded":"2024-03-04T16:27:26.597581+0000","last_fullsized":"2024-03-04T16:27:26.597581+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.981459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.981459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:27.541909+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596192+0000","last_change":"2024-03-04T16:26:34.783238+0000","last_active":"2024-03-04T16:27:26.596192+0000","last_peered":"2024-03-04T16:27:26.596192+0000","last_clean":"2024-03-04T16:27:26.596192+0000","last_became_active":"2024-03-04T16:21:45.437084+0000","last_became_peered":"2024-03-04T16:21:45.437084+0000","last_unstale":"2024-03-04T16:27:26.596192+0000","last_undegraded":"2024-03-04T16:27:26.596192+0000","last_fullsized":"2024-03-04T16:27:26.596192+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:34.783156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:34.783156+0000","last_clean_scrub_stamp":"2024-03-04T16:26:34.783156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:35.437461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":433,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510208+0000","last_change":"2024-03-04T16:26:41.645338+0000","last_active":"2024-03-04T16:27:26.510208+0000","last_peered":"2024-03-04T16:27:26.510208+0000","last_clean":"2024-03-04T16:27:26.510208+0000","last_became_active":"2024-03-04T16:25:35.876577+0000","last_became_peered":"2024-03-04T16:25:35.876577+0000","last_unstale":"2024-03-04T16:27:26.510208+0000","last_undegraded":"2024-03-04T16:27:26.510208+0000","last_fullsized":"2024-03-04T16:27:26.510208+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:41.645255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:26:41.645255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:04.793334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":80,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596980+0000","last_change":"2024-03-04T16:26:30.847294+0000","last_active":"2024-03-04T16:27:26.596980+0000","last_peered":"2024-03-04T16:27:26.596980+0000","last_clean":"2024-03-04T16:27:26.596980+0000","last_became_active":"2024-03-04T16:25:59.530003+0000","last_became_peered":"2024-03-04T16:25:59.530003+0000","last_unstale":"2024-03-04T16:27:26.596980+0000","last_undegraded":"2024-03-04T16:27:26.596980+0000","last_fullsized":"2024-03-04T16:27:26.596980+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-04T16:26:30.847177+0000","last_deep_scrub":"441'26","last_deep_scrub_stamp":"2024-03-04T16:26:30.847177+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.847177+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.800645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":412,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510646+0000","last_change":"2024-03-04T16:26:40.598225+0000","last_active":"2024-03-04T16:27:26.510646+0000","last_peered":"2024-03-04T16:27:26.510646+0000","last_clean":"2024-03-04T16:27:26.510646+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:27:26.510646+0000","last_undegraded":"2024-03-04T16:27:26.510646+0000","last_fullsized":"2024-03-04T16:27:26.510646+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.598141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.598141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:47.374222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1025,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598378+0000","last_change":"2024-03-04T16:26:52.070794+0000","last_active":"2024-03-04T16:27:26.598378+0000","last_peered":"2024-03-04T16:27:26.598378+0000","last_clean":"2024-03-04T16:27:26.598378+0000","last_became_active":"2024-03-04T16:25:41.968664+0000","last_became_peered":"2024-03-04T16:25:41.968664+0000","last_unstale":"2024-03-04T16:27:26.598378+0000","last_undegraded":"2024-03-04T16:27:26.598378+0000","last_fullsized":"2024-03-04T16:27:26.598378+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:52.070677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.070677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.891827+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1331,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596647+0000","last_change":"2024-03-04T16:26:31.917342+0000","last_active":"2024-03-04T16:27:26.596647+0000","last_peered":"2024-03-04T16:27:26.596647+0000","last_clean":"2024-03-04T16:27:26.596647+0000","last_became_active":"2024-03-04T16:21:48.434375+0000","last_became_peered":"2024-03-04T16:21:48.434375+0000","last_unstale":"2024-03-04T16:27:26.596647+0000","last_undegraded":"2024-03-04T16:27:26.596647+0000","last_fullsized":"2024-03-04T16:27:26.596647+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:26:31.917158+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.917158+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:39.727777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":908,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.965728+0000","last_change":"2024-03-04T16:27:14.879085+0000","last_active":"2024-03-04T16:27:24.965728+0000","last_peered":"2024-03-04T16:27:24.965728+0000","last_clean":"2024-03-04T16:27:24.965728+0000","last_became_active":"2024-03-04T16:26:16.731937+0000","last_became_peered":"2024-03-04T16:26:16.731937+0000","last_unstale":"2024-03-04T16:27:24.965728+0000","last_undegraded":"2024-03-04T16:27:24.965728+0000","last_fullsized":"2024-03-04T16:27:24.965728+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:14.878977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:14.878977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.023143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"491'52","reported_seq":389,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598735+0000","last_change":"2024-03-04T16:26:45.826140+0000","last_active":"2024-03-04T16:27:26.598735+0000","last_peered":"2024-03-04T16:27:26.598735+0000","last_clean":"2024-03-04T16:27:26.598735+0000","last_became_active":"2024-03-04T16:25:41.972439+0000","last_became_peered":"2024-03-04T16:25:41.972439+0000","last_unstale":"2024-03-04T16:27:26.598735+0000","last_undegraded":"2024-03-04T16:27:26.598735+0000","last_fullsized":"2024-03-04T16:27:26.598735+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'52","last_scrub_stamp":"2024-03-04T16:26:45.825987+0000","last_deep_scrub":"391'38","last_deep_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_clean_scrub_stamp":"2024-03-04T16:26:45.825987+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:46.472060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597985+0000","last_change":"2024-03-04T16:27:23.891109+0000","last_active":"2024-03-04T16:27:26.597985+0000","last_peered":"2024-03-04T16:27:26.597985+0000","last_clean":"2024-03-04T16:27:26.597985+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:27:26.597985+0000","last_undegraded":"2024-03-04T16:27:26.597985+0000","last_fullsized":"2024-03-04T16:27:26.597985+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:23.891001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:27:23.891001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:53.310100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":68,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596510+0000","last_change":"2024-03-04T16:26:32.781033+0000","last_active":"2024-03-04T16:27:26.596510+0000","last_peered":"2024-03-04T16:27:26.596510+0000","last_clean":"2024-03-04T16:27:26.596510+0000","last_became_active":"2024-03-04T16:25:59.531011+0000","last_became_peered":"2024-03-04T16:25:59.531011+0000","last_unstale":"2024-03-04T16:27:26.596510+0000","last_undegraded":"2024-03-04T16:27:26.596510+0000","last_fullsized":"2024-03-04T16:27:26.596510+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:32.780879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:12.831863+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.780879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:34.448886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":883,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596898+0000","last_change":"2024-03-04T16:26:31.776684+0000","last_active":"2024-03-04T16:27:26.596898+0000","last_peered":"2024-03-04T16:27:26.596898+0000","last_clean":"2024-03-04T16:27:26.596898+0000","last_became_active":"2024-03-04T16:25:41.955219+0000","last_became_peered":"2024-03-04T16:25:41.955219+0000","last_unstale":"2024-03-04T16:27:26.596898+0000","last_undegraded":"2024-03-04T16:27:26.596898+0000","last_fullsized":"2024-03-04T16:27:26.596898+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:31.776609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.776609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:34.783122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":2543,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.888568+0000","last_change":"2024-03-04T16:27:26.888568+0000","last_active":"2024-03-04T16:27:26.888568+0000","last_peered":"2024-03-04T16:27:26.888568+0000","last_clean":"2024-03-04T16:27:26.888568+0000","last_became_active":"2024-03-04T16:25:59.522981+0000","last_became_peered":"2024-03-04T16:25:59.522981+0000","last_unstale":"2024-03-04T16:27:26.888568+0000","last_undegraded":"2024-03-04T16:27:26.888568+0000","last_fullsized":"2024-03-04T16:27:26.888568+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:27:26.888456+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:27:26.888456+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:44.986481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1471,"num_read_kb":1466,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":961,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596713+0000","last_change":"2024-03-04T16:27:08.865537+0000","last_active":"2024-03-04T16:27:26.596713+0000","last_peered":"2024-03-04T16:27:26.596713+0000","last_clean":"2024-03-04T16:27:26.596713+0000","last_became_active":"2024-03-04T16:22:24.838293+0000","last_became_peered":"2024-03-04T16:22:24.838293+0000","last_unstale":"2024-03-04T16:27:26.596713+0000","last_undegraded":"2024-03-04T16:27:26.596713+0000","last_fullsized":"2024-03-04T16:27:26.596713+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-04T16:27:08.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.865437+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.052042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598179+0000","last_change":"2024-03-04T16:26:25.988336+0000","last_active":"2024-03-04T16:27:26.598179+0000","last_peered":"2024-03-04T16:27:26.598179+0000","last_clean":"2024-03-04T16:27:26.598179+0000","last_became_active":"2024-03-04T16:26:25.884588+0000","last_became_peered":"2024-03-04T16:26:25.884588+0000","last_unstale":"2024-03-04T16:27:26.598179+0000","last_undegraded":"2024-03-04T16:27:26.598179+0000","last_fullsized":"2024-03-04T16:27:26.598179+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:25.988227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:25.988227+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.988227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:49.637237+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596040+0000","last_change":"2024-03-04T16:26:46.757066+0000","last_active":"2024-03-04T16:27:26.596040+0000","last_peered":"2024-03-04T16:27:26.596040+0000","last_clean":"2024-03-04T16:27:26.596040+0000","last_became_active":"2024-03-04T16:21:45.442233+0000","last_became_peered":"2024-03-04T16:21:45.442233+0000","last_unstale":"2024-03-04T16:27:26.596040+0000","last_undegraded":"2024-03-04T16:27:26.596040+0000","last_fullsized":"2024-03-04T16:27:26.596040+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:46.756942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:35.788869+0000","last_clean_scrub_stamp":"2024-03-04T16:26:46.756942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:51.824085+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596670+0000","last_change":"2024-03-04T16:26:06.823971+0000","last_active":"2024-03-04T16:27:26.596670+0000","last_peered":"2024-03-04T16:27:26.596670+0000","last_clean":"2024-03-04T16:27:26.596670+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:27:26.596670+0000","last_undegraded":"2024-03-04T16:27:26.596670+0000","last_fullsized":"2024-03-04T16:27:26.596670+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:06.823861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:06.823861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'47","reported_seq":890,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596308+0000","last_change":"2024-03-04T16:26:32.782094+0000","last_active":"2024-03-04T16:27:26.596308+0000","last_peered":"2024-03-04T16:27:26.596308+0000","last_clean":"2024-03-04T16:27:26.596308+0000","last_became_active":"2024-03-04T16:25:56.057569+0000","last_became_peered":"2024-03-04T16:25:56.057569+0000","last_unstale":"2024-03-04T16:27:26.596308+0000","last_undegraded":"2024-03-04T16:27:26.596308+0000","last_fullsized":"2024-03-04T16:27:26.596308+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'47","last_scrub_stamp":"2024-03-04T16:26:32.782005+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.782005+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.396594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597522+0000","last_change":"2024-03-04T16:27:17.611404+0000","last_active":"2024-03-04T16:27:26.597522+0000","last_peered":"2024-03-04T16:27:26.597522+0000","last_clean":"2024-03-04T16:27:26.597522+0000","last_became_active":"2024-03-04T16:21:45.412152+0000","last_became_peered":"2024-03-04T16:21:45.412152+0000","last_unstale":"2024-03-04T16:27:26.597522+0000","last_undegraded":"2024-03-04T16:27:26.597522+0000","last_fullsized":"2024-03-04T16:27:26.597522+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:17.611297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.611297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.488811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1034,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596014+0000","last_change":"2024-03-04T16:26:02.724516+0000","last_active":"2024-03-04T16:27:26.596014+0000","last_peered":"2024-03-04T16:27:26.596014+0000","last_clean":"2024-03-04T16:27:26.596014+0000","last_became_active":"2024-03-04T16:25:35.905478+0000","last_became_peered":"2024-03-04T16:25:35.905478+0000","last_unstale":"2024-03-04T16:27:26.596014+0000","last_undegraded":"2024-03-04T16:27:26.596014+0000","last_fullsized":"2024-03-04T16:27:26.596014+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:02.724383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:02.724383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:28.900944+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598335+0000","last_change":"2024-03-04T16:27:08.113585+0000","last_active":"2024-03-04T16:27:26.598335+0000","last_peered":"2024-03-04T16:27:26.598335+0000","last_clean":"2024-03-04T16:27:26.598335+0000","last_became_active":"2024-03-04T16:25:35.915319+0000","last_became_peered":"2024-03-04T16:25:35.915319+0000","last_unstale":"2024-03-04T16:27:26.598335+0000","last_undegraded":"2024-03-04T16:27:26.598335+0000","last_fullsized":"2024-03-04T16:27:26.598335+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:08.113444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.113444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:22.295481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":98,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596512+0000","last_change":"2024-03-04T16:26:33.773431+0000","last_active":"2024-03-04T16:27:26.596512+0000","last_peered":"2024-03-04T16:27:26.596512+0000","last_clean":"2024-03-04T16:27:26.596512+0000","last_became_active":"2024-03-04T16:26:25.883379+0000","last_became_peered":"2024-03-04T16:26:25.883379+0000","last_unstale":"2024-03-04T16:27:26.596512+0000","last_undegraded":"2024-03-04T16:27:26.596512+0000","last_fullsized":"2024-03-04T16:27:26.596512+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:26:33.773330+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:26:33.773330+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:50.397293+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"494'13","reported_seq":164,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596243+0000","last_change":"2024-03-04T16:26:19.742627+0000","last_active":"2024-03-04T16:27:26.596243+0000","last_peered":"2024-03-04T16:27:26.596243+0000","last_clean":"2024-03-04T16:27:26.596243+0000","last_became_active":"2024-03-04T16:25:56.057430+0000","last_became_peered":"2024-03-04T16:25:56.057430+0000","last_unstale":"2024-03-04T16:27:26.596243+0000","last_undegraded":"2024-03-04T16:27:26.596243+0000","last_fullsized":"2024-03-04T16:27:26.596243+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'13","last_scrub_stamp":"2024-03-04T16:26:19.742509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.742509+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:33.843176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598278+0000","last_change":"2024-03-04T16:27:03.027523+0000","last_active":"2024-03-04T16:27:26.598278+0000","last_peered":"2024-03-04T16:27:26.598278+0000","last_clean":"2024-03-04T16:27:26.598278+0000","last_became_active":"2024-03-04T16:21:45.413068+0000","last_became_peered":"2024-03-04T16:21:45.413068+0000","last_unstale":"2024-03-04T16:27:26.598278+0000","last_undegraded":"2024-03-04T16:27:26.598278+0000","last_fullsized":"2024-03-04T16:27:26.598278+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:03.027421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:27:03.027421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.685129+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"505'2","reported_seq":876,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596809+0000","last_change":"2024-03-04T16:26:16.707720+0000","last_active":"2024-03-04T16:27:26.596809+0000","last_peered":"2024-03-04T16:27:26.596809+0000","last_clean":"2024-03-04T16:27:26.596809+0000","last_became_active":"2024-03-04T16:25:41.956066+0000","last_became_peered":"2024-03-04T16:25:41.956066+0000","last_unstale":"2024-03-04T16:27:26.596809+0000","last_undegraded":"2024-03-04T16:27:26.596809+0000","last_fullsized":"2024-03-04T16:27:26.596809+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'2","last_scrub_stamp":"2024-03-04T16:26:16.707585+0000","last_deep_scrub":"505'2","last_deep_scrub_stamp":"2024-03-04T16:26:16.707585+0000","last_clean_scrub_stamp":"2024-03-04T16:26:16.707585+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.345591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'49","reported_seq":94,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511487+0000","last_change":"2024-03-04T16:27:00.740309+0000","last_active":"2024-03-04T16:27:26.511487+0000","last_peered":"2024-03-04T16:27:26.511487+0000","last_clean":"2024-03-04T16:27:26.511487+0000","last_became_active":"2024-03-04T16:25:41.963039+0000","last_became_peered":"2024-03-04T16:25:41.963039+0000","last_unstale":"2024-03-04T16:27:26.511487+0000","last_undegraded":"2024-03-04T16:27:26.511487+0000","last_fullsized":"2024-03-04T16:27:26.511487+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'49","last_scrub_stamp":"2024-03-04T16:27:00.740174+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.740174+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:04.779080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"517'13","reported_seq":125,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.965020+0000","last_change":"2024-03-04T16:26:55.823122+0000","last_active":"2024-03-04T16:27:24.965020+0000","last_peered":"2024-03-04T16:27:24.965020+0000","last_clean":"2024-03-04T16:27:24.965020+0000","last_became_active":"2024-03-04T16:25:41.945789+0000","last_became_peered":"2024-03-04T16:25:41.945789+0000","last_unstale":"2024-03-04T16:27:24.965020+0000","last_undegraded":"2024-03-04T16:27:24.965020+0000","last_fullsized":"2024-03-04T16:27:24.965020+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'13","last_scrub_stamp":"2024-03-04T16:26:55.823038+0000","last_deep_scrub":"517'13","last_deep_scrub_stamp":"2024-03-04T16:26:55.823038+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.823038+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.287679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511449+0000","last_change":"2024-03-04T16:26:18.781428+0000","last_active":"2024-03-04T16:27:26.511449+0000","last_peered":"2024-03-04T16:27:26.511449+0000","last_clean":"2024-03-04T16:27:26.511449+0000","last_became_active":"2024-03-04T16:25:41.965545+0000","last_became_peered":"2024-03-04T16:25:41.965545+0000","last_unstale":"2024-03-04T16:27:26.511449+0000","last_undegraded":"2024-03-04T16:27:26.511449+0000","last_fullsized":"2024-03-04T16:27:26.511449+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:18.781302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:18.781302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.039612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":432,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966919+0000","last_change":"2024-03-04T16:26:48.837306+0000","last_active":"2024-03-04T16:27:24.966919+0000","last_peered":"2024-03-04T16:27:24.966919+0000","last_clean":"2024-03-04T16:27:24.966919+0000","last_became_active":"2024-03-04T16:26:16.732380+0000","last_became_peered":"2024-03-04T16:26:16.732380+0000","last_unstale":"2024-03-04T16:27:24.966919+0000","last_undegraded":"2024-03-04T16:27:24.966919+0000","last_fullsized":"2024-03-04T16:27:24.966919+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:48.837217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.837217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.108872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'42","reported_seq":106,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.964797+0000","last_change":"2024-03-04T16:27:06.921399+0000","last_active":"2024-03-04T16:27:24.964797+0000","last_peered":"2024-03-04T16:27:24.964797+0000","last_clean":"2024-03-04T16:27:24.964797+0000","last_became_active":"2024-03-04T16:25:35.913673+0000","last_became_peered":"2024-03-04T16:25:35.913673+0000","last_unstale":"2024-03-04T16:27:24.964797+0000","last_undegraded":"2024-03-04T16:27:24.964797+0000","last_fullsized":"2024-03-04T16:27:24.964797+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-04T16:27:06.921303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.921303+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:19.081502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511304+0000","last_change":"2024-03-04T16:26:57.714648+0000","last_active":"2024-03-04T16:27:26.511304+0000","last_peered":"2024-03-04T16:27:26.511304+0000","last_clean":"2024-03-04T16:27:26.511304+0000","last_became_active":"2024-03-04T16:25:41.963328+0000","last_became_peered":"2024-03-04T16:25:41.963328+0000","last_unstale":"2024-03-04T16:27:26.511304+0000","last_undegraded":"2024-03-04T16:27:26.511304+0000","last_fullsized":"2024-03-04T16:27:26.511304+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:57.714536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:57.714536+0000","last_clean_scrub_stamp":"2024-03-04T16:26:57.714536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.664488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598082+0000","last_change":"2024-03-04T16:26:38.121206+0000","last_active":"2024-03-04T16:27:26.598082+0000","last_peered":"2024-03-04T16:27:26.598082+0000","last_clean":"2024-03-04T16:27:26.598082+0000","last_became_active":"2024-03-04T16:25:59.527386+0000","last_became_peered":"2024-03-04T16:25:59.527386+0000","last_unstale":"2024-03-04T16:27:26.598082+0000","last_undegraded":"2024-03-04T16:27:26.598082+0000","last_fullsized":"2024-03-04T16:27:26.598082+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:38.121069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:38.121069+0000","last_clean_scrub_stamp":"2024-03-04T16:26:38.121069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.746919+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"491'10","reported_seq":103,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512305+0000","last_change":"2024-03-04T16:26:55.718528+0000","last_active":"2024-03-04T16:27:26.512305+0000","last_peered":"2024-03-04T16:27:26.512305+0000","last_clean":"2024-03-04T16:27:26.512305+0000","last_became_active":"2024-03-04T16:25:41.961001+0000","last_became_peered":"2024-03-04T16:25:41.961001+0000","last_unstale":"2024-03-04T16:27:26.512305+0000","last_undegraded":"2024-03-04T16:27:26.512305+0000","last_fullsized":"2024-03-04T16:27:26.512305+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'10","last_scrub_stamp":"2024-03-04T16:26:55.718433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.718433+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:05.895715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":80,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.964916+0000","last_change":"2024-03-04T16:26:10.736176+0000","last_active":"2024-03-04T16:27:24.964916+0000","last_peered":"2024-03-04T16:27:24.964916+0000","last_clean":"2024-03-04T16:27:24.964916+0000","last_became_active":"2024-03-04T16:25:35.905845+0000","last_became_peered":"2024-03-04T16:25:35.905845+0000","last_unstale":"2024-03-04T16:27:24.964916+0000","last_undegraded":"2024-03-04T16:27:24.964916+0000","last_fullsized":"2024-03-04T16:27:24.964916+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:10.736056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:10.736056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'46","reported_seq":1157,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597084+0000","last_change":"2024-03-04T16:26:52.768796+0000","last_active":"2024-03-04T16:27:26.597084+0000","last_peered":"2024-03-04T16:27:26.597084+0000","last_clean":"2024-03-04T16:27:26.597084+0000","last_became_active":"2024-03-04T16:25:41.958609+0000","last_became_peered":"2024-03-04T16:25:41.958609+0000","last_unstale":"2024-03-04T16:27:26.597084+0000","last_undegraded":"2024-03-04T16:27:26.597084+0000","last_fullsized":"2024-03-04T16:27:26.597084+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:26:52.768648+0000","last_deep_scrub":"441'46","last_deep_scrub_stamp":"2024-03-04T16:25:18.623813+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.768648+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:06.856507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"507'2","reported_seq":248,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.967065+0000","last_change":"2024-03-04T16:26:20.703392+0000","last_active":"2024-03-04T16:27:24.967065+0000","last_peered":"2024-03-04T16:27:24.967065+0000","last_clean":"2024-03-04T16:27:24.967065+0000","last_became_active":"2024-03-04T16:26:16.733440+0000","last_became_peered":"2024-03-04T16:26:16.733440+0000","last_unstale":"2024-03-04T16:27:24.967065+0000","last_undegraded":"2024-03-04T16:27:24.967065+0000","last_fullsized":"2024-03-04T16:27:24.967065+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-03-04T16:26:20.703307+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-03-04T16:26:20.703307+0000","last_clean_scrub_stamp":"2024-03-04T16:26:20.703307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:48.454458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598752+0000","last_change":"2024-03-04T16:27:07.064135+0000","last_active":"2024-03-04T16:27:26.598752+0000","last_peered":"2024-03-04T16:27:26.598752+0000","last_clean":"2024-03-04T16:27:26.598752+0000","last_became_active":"2024-03-04T16:25:41.969123+0000","last_became_peered":"2024-03-04T16:25:41.969123+0000","last_unstale":"2024-03-04T16:27:26.598752+0000","last_undegraded":"2024-03-04T16:27:26.598752+0000","last_fullsized":"2024-03-04T16:27:26.598752+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.064036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:07.064036+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.064036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:20.800314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511424+0000","last_change":"2024-03-04T16:26:56.709477+0000","last_active":"2024-03-04T16:27:26.511424+0000","last_peered":"2024-03-04T16:27:26.511424+0000","last_clean":"2024-03-04T16:27:26.511424+0000","last_became_active":"2024-03-04T16:25:41.966246+0000","last_became_peered":"2024-03-04T16:25:41.966246+0000","last_unstale":"2024-03-04T16:27:26.511424+0000","last_undegraded":"2024-03-04T16:27:26.511424+0000","last_fullsized":"2024-03-04T16:27:26.511424+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:56.709331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:56.709331+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.709331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.623820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'32","reported_seq":108,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512177+0000","last_change":"2024-03-04T16:26:29.705756+0000","last_active":"2024-03-04T16:27:26.512177+0000","last_peered":"2024-03-04T16:27:26.512177+0000","last_clean":"2024-03-04T16:27:26.512177+0000","last_became_active":"2024-03-04T16:25:59.525685+0000","last_became_peered":"2024-03-04T16:25:59.525685+0000","last_unstale":"2024-03-04T16:27:26.512177+0000","last_undegraded":"2024-03-04T16:27:26.512177+0000","last_fullsized":"2024-03-04T16:27:26.512177+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:26:29.705574+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-04T16:26:29.705574+0000","last_clean_scrub_stamp":"2024-03-04T16:26:29.705574+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:58.114809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":456,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510547+0000","last_change":"2024-03-04T16:26:59.032019+0000","last_active":"2024-03-04T16:27:26.510547+0000","last_peered":"2024-03-04T16:27:26.510547+0000","last_clean":"2024-03-04T16:27:26.510547+0000","last_became_active":"2024-03-04T16:25:41.958623+0000","last_became_peered":"2024-03-04T16:25:41.958623+0000","last_unstale":"2024-03-04T16:27:26.510547+0000","last_undegraded":"2024-03-04T16:27:26.510547+0000","last_fullsized":"2024-03-04T16:27:26.510547+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:59.031905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:32.684717+0000","last_clean_scrub_stamp":"2024-03-04T16:26:59.031905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:19.598491+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1023,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596738+0000","last_change":"2024-03-04T16:26:48.646715+0000","last_active":"2024-03-04T16:27:26.596738+0000","last_peered":"2024-03-04T16:27:26.596738+0000","last_clean":"2024-03-04T16:27:26.596738+0000","last_became_active":"2024-03-04T16:25:41.957803+0000","last_became_peered":"2024-03-04T16:25:41.957803+0000","last_unstale":"2024-03-04T16:27:26.596738+0000","last_undegraded":"2024-03-04T16:27:26.596738+0000","last_fullsized":"2024-03-04T16:27:26.596738+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:48.646572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:48.646572+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.646572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.653663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":299,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597840+0000","last_change":"2024-03-04T16:27:22.631026+0000","last_active":"2024-03-04T16:27:26.597840+0000","last_peered":"2024-03-04T16:27:26.597840+0000","last_clean":"2024-03-04T16:27:26.597840+0000","last_became_active":"2024-03-04T16:25:59.526640+0000","last_became_peered":"2024-03-04T16:25:59.526640+0000","last_unstale":"2024-03-04T16:27:26.597840+0000","last_undegraded":"2024-03-04T16:27:26.597840+0000","last_fullsized":"2024-03-04T16:27:26.597840+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:22.630920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:27:22.630920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.299972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":96,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512234+0000","last_change":"2024-03-04T16:26:32.725623+0000","last_active":"2024-03-04T16:27:26.512234+0000","last_peered":"2024-03-04T16:27:26.512234+0000","last_clean":"2024-03-04T16:27:26.512234+0000","last_became_active":"2024-03-04T16:25:41.960919+0000","last_became_peered":"2024-03-04T16:25:41.960919+0000","last_unstale":"2024-03-04T16:27:26.512234+0000","last_undegraded":"2024-03-04T16:27:26.512234+0000","last_fullsized":"2024-03-04T16:27:26.512234+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-04T16:26:32.725509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.725509+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.100907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512274+0000","last_change":"2024-03-04T16:27:07.865879+0000","last_active":"2024-03-04T16:27:26.512274+0000","last_peered":"2024-03-04T16:27:26.512274+0000","last_clean":"2024-03-04T16:27:26.512274+0000","last_became_active":"2024-03-04T16:25:59.522487+0000","last_became_peered":"2024-03-04T16:25:59.522487+0000","last_unstale":"2024-03-04T16:27:26.512274+0000","last_undegraded":"2024-03-04T16:27:26.512274+0000","last_fullsized":"2024-03-04T16:27:26.512274+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.865756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:56.691335+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.865756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.941362+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":517,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510636+0000","last_change":"2024-03-04T16:26:36.698181+0000","last_active":"2024-03-04T16:27:26.510636+0000","last_peered":"2024-03-04T16:27:26.510636+0000","last_clean":"2024-03-04T16:27:26.510636+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:27:26.510636+0000","last_undegraded":"2024-03-04T16:27:26.510636+0000","last_fullsized":"2024-03-04T16:27:26.510636+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:26:36.698080+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:26:36.698080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.703568+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":128,"reported_epoch":520,"state":"active+clean","last_fresh":"2024-03-04T16:27:24.966339+0000","last_change":"2024-03-04T16:26:41.754214+0000","last_active":"2024-03-04T16:27:24.966339+0000","last_peered":"2024-03-04T16:27:24.966339+0000","last_clean":"2024-03-04T16:27:24.966339+0000","last_became_active":"2024-03-04T16:26:16.733370+0000","last_became_peered":"2024-03-04T16:26:16.733370+0000","last_unstale":"2024-03-04T16:27:24.966339+0000","last_undegraded":"2024-03-04T16:27:24.966339+0000","last_fullsized":"2024-03-04T16:27:24.966339+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:41.754075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:41.754075+0000","last_clean_scrub_stamp":"2024-03-04T16:26:41.754075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:02.504263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511736+0000","last_change":"2024-03-04T16:26:25.764118+0000","last_active":"2024-03-04T16:27:26.511736+0000","last_peered":"2024-03-04T16:27:26.511736+0000","last_clean":"2024-03-04T16:27:26.511736+0000","last_became_active":"2024-03-04T16:25:41.957322+0000","last_became_peered":"2024-03-04T16:25:41.957322+0000","last_unstale":"2024-03-04T16:27:26.511736+0000","last_undegraded":"2024-03-04T16:27:26.511736+0000","last_fullsized":"2024-03-04T16:27:26.511736+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:26:25.764002+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.764002+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:50.875243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1471,"num_read_kb":1466,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":4137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1519,"num_read_kb":1276,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":14974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1058,"ondisk_log_size":1058,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":779,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070766,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957196,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92787956,"statfs":{"total":95995035648,"available":95014866944,"internally_reserved":0,"allocated":1441792,"data_stored":1014437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":978713034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365106,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433784,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93311368,"statfs":{"total":95995035648,"available":95550840832,"internally_reserved":0,"allocated":1417216,"data_stored":1012125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128804,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":1127934,"kb_avail":92616348,"statfs":{"total":95995035648,"available":94839140352,"internally_reserved":0,"allocated":872448,"data_stored":423199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1155004869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738704,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925260,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92819892,"statfs":{"total":95995035648,"available":95047569408,"internally_reserved":0,"allocated":1441792,"data_stored":1013781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411654,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073156,"statfs":{"total":95995035648,"available":95306911744,"internally_reserved":0,"allocated":831488,"data_stored":422587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150016,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202684,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":1201854,"kb_avail":92542468,"statfs":{"total":95995035648,"available":94763487232,"internally_reserved":0,"allocated":839680,"data_stored":410085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1230698952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953788,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740040,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005112,"statfs":{"total":95995035648,"available":95237234688,"internally_reserved":0,"allocated":847872,"data_stored":422637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626744,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032952,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":92712200,"statfs":{"total":95995035648,"available":94937292800,"internally_reserved":0,"allocated":831488,"data_stored":422193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1056897478},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:27:30.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-04T16:27:31.060 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[29779]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[29779]: from='client.36341 v1:172.21.15.38:0/1017544158' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[29779]: 3.4 deep-scrub starts 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[29779]: 3.4 deep-scrub ok 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[38113]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[38113]: from='client.36341 v1:172.21.15.38:0/1017544158' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[38113]: 3.4 deep-scrub starts 2024-03-04T16:27:31.846 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:31 smithi038 ceph-mon[38113]: 3.4 deep-scrub ok 2024-03-04T16:27:31.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:31 smithi176 ceph-mon[33619]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-04T16:27:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:31 smithi176 ceph-mon[33619]: from='client.36341 v1:172.21.15.38:0/1017544158' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:31 smithi176 ceph-mon[33619]: 3.4 deep-scrub starts 2024-03-04T16:27:31.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:31 smithi176 ceph-mon[33619]: 3.4 deep-scrub ok 2024-03-04T16:27:32.726 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:32 smithi038 ceph-mon[29779]: 6.0 scrub starts 2024-03-04T16:27:32.726 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:32 smithi038 ceph-mon[29779]: 6.0 scrub ok 2024-03-04T16:27:32.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:32 smithi038 ceph-mon[38113]: 6.0 scrub starts 2024-03-04T16:27:32.726 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:32 smithi038 ceph-mon[38113]: 6.0 scrub ok 2024-03-04T16:27:32.857 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:32 smithi176 ceph-mon[33619]: 6.0 scrub starts 2024-03-04T16:27:32.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:32 smithi176 ceph-mon[33619]: 6.0 scrub ok 2024-03-04T16:27:32.905 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-04T16:27:32.907 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-04T16:27:33.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[29779]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:33.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[29779]: from='client.36347 v1:172.21.15.38:0/746183218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:33.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[29779]: 5.15 scrub starts 2024-03-04T16:27:33.649 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[29779]: 5.15 scrub ok 2024-03-04T16:27:33.649 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: ::ffff:172.21.15.176 - - [04/Mar/2024:16:27:33] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-04T16:27:33.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[38113]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:33.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[38113]: from='client.36347 v1:172.21.15.38:0/746183218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:33.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[38113]: 5.15 scrub starts 2024-03-04T16:27:33.650 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:33 smithi038 ceph-mon[38113]: 5.15 scrub ok 2024-03-04T16:27:33.675 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":1169,"stamp":"2024-03-04T16:27:31.814296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3219,"num_read_kb":3065,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":781,"num_bytes_recovered":4153671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1309,"ondisk_log_size":1309,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7092748,"kb_used_data":8348,"kb_used_omap":12,"kb_used_meta":7084275,"kb_avail":742868468,"statfs":{"total":767960285184,"available":760697311232,"internally_reserved":0,"allocated":8548352,"data_stored":5154482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":7254298162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002461"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511729+0000","last_change":"2024-03-04T16:27:03.801816+0000","last_active":"2024-03-04T16:27:26.511729+0000","last_peered":"2024-03-04T16:27:26.511729+0000","last_clean":"2024-03-04T16:27:26.511729+0000","last_became_active":"2024-03-04T16:25:59.520911+0000","last_became_peered":"2024-03-04T16:25:59.520911+0000","last_unstale":"2024-03-04T16:27:26.511729+0000","last_undegraded":"2024-03-04T16:27:26.511729+0000","last_fullsized":"2024-03-04T16:27:26.511729+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:03.801685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:03.801685+0000","last_clean_scrub_stamp":"2024-03-04T16:27:03.801685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.017193+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":997,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595933+0000","last_change":"2024-03-04T16:27:07.790924+0000","last_active":"2024-03-04T16:27:26.595933+0000","last_peered":"2024-03-04T16:27:26.595933+0000","last_clean":"2024-03-04T16:27:26.595933+0000","last_became_active":"2024-03-04T16:02:47.767427+0000","last_became_peered":"2024-03-04T16:02:47.767427+0000","last_unstale":"2024-03-04T16:27:26.595933+0000","last_undegraded":"2024-03-04T16:27:26.595933+0000","last_fullsized":"2024-03-04T16:27:26.595933+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.790846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:40.696222+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.790846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.927480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":869,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596682+0000","last_change":"2024-03-04T16:26:57.875935+0000","last_active":"2024-03-04T16:27:26.596682+0000","last_peered":"2024-03-04T16:27:26.596682+0000","last_clean":"2024-03-04T16:27:26.596682+0000","last_became_active":"2024-03-04T16:19:04.826698+0000","last_became_peered":"2024-03-04T16:19:04.826698+0000","last_unstale":"2024-03-04T16:27:26.596682+0000","last_undegraded":"2024-03-04T16:27:26.596682+0000","last_fullsized":"2024-03-04T16:27:26.596682+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:57.875855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:13.597504+0000","last_clean_scrub_stamp":"2024-03-04T16:26:57.875855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:18.783757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511083+0000","last_change":"2024-03-04T16:26:54.693305+0000","last_active":"2024-03-04T16:27:26.511083+0000","last_peered":"2024-03-04T16:27:26.511083+0000","last_clean":"2024-03-04T16:27:26.511083+0000","last_became_active":"2024-03-04T16:25:41.956367+0000","last_became_peered":"2024-03-04T16:25:41.956367+0000","last_unstale":"2024-03-04T16:27:26.511083+0000","last_undegraded":"2024-03-04T16:27:26.511083+0000","last_fullsized":"2024-03-04T16:27:26.511083+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:54.693176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.693176+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:15.427563+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512766+0000","last_change":"2024-03-04T16:27:10.127287+0000","last_active":"2024-03-04T16:27:26.512766+0000","last_peered":"2024-03-04T16:27:26.512766+0000","last_clean":"2024-03-04T16:27:26.512766+0000","last_became_active":"2024-03-04T16:25:35.906317+0000","last_became_peered":"2024-03-04T16:25:35.906317+0000","last_unstale":"2024-03-04T16:27:26.512766+0000","last_undegraded":"2024-03-04T16:27:26.512766+0000","last_fullsized":"2024-03-04T16:27:26.512766+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:10.127173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:46.313300+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.127173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.871370+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511166+0000","last_change":"2024-03-04T16:26:50.816244+0000","last_active":"2024-03-04T16:27:26.511166+0000","last_peered":"2024-03-04T16:27:26.511166+0000","last_clean":"2024-03-04T16:27:26.511166+0000","last_became_active":"2024-03-04T16:25:41.956943+0000","last_became_peered":"2024-03-04T16:25:41.956943+0000","last_unstale":"2024-03-04T16:27:26.511166+0000","last_undegraded":"2024-03-04T16:27:26.511166+0000","last_fullsized":"2024-03-04T16:27:26.511166+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:50.816139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:13.629592+0000","last_clean_scrub_stamp":"2024-03-04T16:26:50.816139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:15.755176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":82,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512959+0000","last_change":"2024-03-04T16:26:26.735107+0000","last_active":"2024-03-04T16:27:26.512959+0000","last_peered":"2024-03-04T16:27:26.512959+0000","last_clean":"2024-03-04T16:27:26.512959+0000","last_became_active":"2024-03-04T16:25:35.901847+0000","last_became_peered":"2024-03-04T16:25:35.901847+0000","last_unstale":"2024-03-04T16:27:26.512959+0000","last_undegraded":"2024-03-04T16:27:26.512959+0000","last_fullsized":"2024-03-04T16:27:26.512959+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.735010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:23:08.925783+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.735010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:37.980069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'39","reported_seq":997,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512515+0000","last_change":"2024-03-04T16:26:30.685418+0000","last_active":"2024-03-04T16:27:26.512515+0000","last_peered":"2024-03-04T16:27:26.512515+0000","last_clean":"2024-03-04T16:27:26.512515+0000","last_became_active":"2024-03-04T16:26:16.730442+0000","last_became_peered":"2024-03-04T16:26:16.730442+0000","last_unstale":"2024-03-04T16:27:26.512515+0000","last_undegraded":"2024-03-04T16:27:26.512515+0000","last_fullsized":"2024-03-04T16:27:26.512515+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:26:30.685312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.685312+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.648562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":850,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595872+0000","last_change":"2024-03-04T16:26:29.845038+0000","last_active":"2024-03-04T16:27:26.595872+0000","last_peered":"2024-03-04T16:27:26.595872+0000","last_clean":"2024-03-04T16:27:26.595872+0000","last_became_active":"2024-03-04T16:07:36.224517+0000","last_became_peered":"2024-03-04T16:07:36.224517+0000","last_unstale":"2024-03-04T16:27:26.595872+0000","last_undegraded":"2024-03-04T16:27:26.595872+0000","last_fullsized":"2024-03-04T16:27:26.595872+0000","mapping_epoch":140,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":141,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:29.844928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:12.891748+0000","last_clean_scrub_stamp":"2024-03-04T16:26:29.844928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:36.972512+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":989,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597041+0000","last_change":"2024-03-04T16:26:19.061141+0000","last_active":"2024-03-04T16:27:26.597041+0000","last_peered":"2024-03-04T16:27:26.597041+0000","last_clean":"2024-03-04T16:27:26.597041+0000","last_became_active":"2024-03-04T16:02:47.775067+0000","last_became_peered":"2024-03-04T16:02:47.775067+0000","last_unstale":"2024-03-04T16:27:26.597041+0000","last_undegraded":"2024-03-04T16:27:26.597041+0000","last_fullsized":"2024-03-04T16:27:26.597041+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.061034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:06.888592+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.061034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598188+0000","last_change":"2024-03-04T16:27:20.063727+0000","last_active":"2024-03-04T16:27:26.598188+0000","last_peered":"2024-03-04T16:27:26.598188+0000","last_clean":"2024-03-04T16:27:26.598188+0000","last_became_active":"2024-03-04T16:25:41.969493+0000","last_became_peered":"2024-03-04T16:25:41.969493+0000","last_unstale":"2024-03-04T16:27:26.598188+0000","last_undegraded":"2024-03-04T16:27:26.598188+0000","last_fullsized":"2024-03-04T16:27:26.598188+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:20.063653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:20.063653+0000","last_clean_scrub_stamp":"2024-03-04T16:27:20.063653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:40.102745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'25","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511734+0000","last_change":"2024-03-04T16:27:08.824922+0000","last_active":"2024-03-04T16:27:26.511734+0000","last_peered":"2024-03-04T16:27:26.511734+0000","last_clean":"2024-03-04T16:27:26.511734+0000","last_became_active":"2024-03-04T16:25:41.956644+0000","last_became_peered":"2024-03-04T16:25:41.956644+0000","last_unstale":"2024-03-04T16:27:26.511734+0000","last_undegraded":"2024-03-04T16:27:26.511734+0000","last_fullsized":"2024-03-04T16:27:26.511734+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:27:08.824811+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:08:22.740850+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.824811+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.204239+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596426+0000","last_change":"2024-03-04T16:27:21.132837+0000","last_active":"2024-03-04T16:27:26.596426+0000","last_peered":"2024-03-04T16:27:26.596426+0000","last_clean":"2024-03-04T16:27:26.596426+0000","last_became_active":"2024-03-04T16:26:25.882740+0000","last_became_peered":"2024-03-04T16:26:25.882740+0000","last_unstale":"2024-03-04T16:27:26.596426+0000","last_undegraded":"2024-03-04T16:27:26.596426+0000","last_fullsized":"2024-03-04T16:27:26.596426+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:21.132730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:09.872152+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.132730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.971036+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1018,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597061+0000","last_change":"2024-03-04T16:26:56.982294+0000","last_active":"2024-03-04T16:27:26.597061+0000","last_peered":"2024-03-04T16:27:26.597061+0000","last_clean":"2024-03-04T16:27:26.597061+0000","last_became_active":"2024-03-04T16:25:41.959358+0000","last_became_peered":"2024-03-04T16:25:41.959358+0000","last_unstale":"2024-03-04T16:27:26.597061+0000","last_undegraded":"2024-03-04T16:27:26.597061+0000","last_fullsized":"2024-03-04T16:27:26.597061+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:56.982182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:39.966717+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.982182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.940084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1029,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597397+0000","last_change":"2024-03-04T16:27:14.138515+0000","last_active":"2024-03-04T16:27:26.597397+0000","last_peered":"2024-03-04T16:27:26.597397+0000","last_clean":"2024-03-04T16:27:26.597397+0000","last_became_active":"2024-03-04T16:25:41.969615+0000","last_became_peered":"2024-03-04T16:25:41.969615+0000","last_unstale":"2024-03-04T16:27:26.597397+0000","last_undegraded":"2024-03-04T16:27:26.597397+0000","last_fullsized":"2024-03-04T16:27:26.597397+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:14.138418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:14.138418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.285858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"441'11","reported_seq":104,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511917+0000","last_change":"2024-03-04T16:27:17.842174+0000","last_active":"2024-03-04T16:27:26.511917+0000","last_peered":"2024-03-04T16:27:26.511917+0000","last_clean":"2024-03-04T16:27:26.511917+0000","last_became_active":"2024-03-04T16:25:59.529024+0000","last_became_peered":"2024-03-04T16:25:59.529024+0000","last_unstale":"2024-03-04T16:27:26.511917+0000","last_undegraded":"2024-03-04T16:27:26.511917+0000","last_fullsized":"2024-03-04T16:27:26.511917+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'11","last_scrub_stamp":"2024-03-04T16:27:17.842067+0000","last_deep_scrub":"441'11","last_deep_scrub_stamp":"2024-03-04T16:26:10.685697+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.842067+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:22.385108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512167+0000","last_change":"2024-03-04T16:26:51.856186+0000","last_active":"2024-03-04T16:27:26.512167+0000","last_peered":"2024-03-04T16:27:26.512167+0000","last_clean":"2024-03-04T16:27:26.512167+0000","last_became_active":"2024-03-04T16:25:41.963647+0000","last_became_peered":"2024-03-04T16:25:41.963647+0000","last_unstale":"2024-03-04T16:27:26.512167+0000","last_undegraded":"2024-03-04T16:27:26.512167+0000","last_fullsized":"2024-03-04T16:27:26.512167+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:51.856082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:51.856082+0000","last_clean_scrub_stamp":"2024-03-04T16:26:51.856082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:13.230502+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597041+0000","last_change":"2024-03-04T16:26:55.771511+0000","last_active":"2024-03-04T16:27:26.597041+0000","last_peered":"2024-03-04T16:27:26.597041+0000","last_clean":"2024-03-04T16:27:26.597041+0000","last_became_active":"2024-03-04T16:25:41.958642+0000","last_became_peered":"2024-03-04T16:25:41.958642+0000","last_unstale":"2024-03-04T16:27:26.597041+0000","last_undegraded":"2024-03-04T16:27:26.597041+0000","last_fullsized":"2024-03-04T16:27:26.597041+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:55.771436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.771436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:16.232921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":873,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596336+0000","last_change":"2024-03-04T16:27:25.879064+0000","last_active":"2024-03-04T16:27:26.596336+0000","last_peered":"2024-03-04T16:27:26.596336+0000","last_clean":"2024-03-04T16:27:26.596336+0000","last_became_active":"2024-03-04T16:25:41.955200+0000","last_became_peered":"2024-03-04T16:25:41.955200+0000","last_unstale":"2024-03-04T16:27:26.596336+0000","last_undegraded":"2024-03-04T16:27:26.596336+0000","last_fullsized":"2024-03-04T16:27:26.596336+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:25.878983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:25.878983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:40.924025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":490,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510527+0000","last_change":"2024-03-04T16:26:47.760763+0000","last_active":"2024-03-04T16:27:26.510527+0000","last_peered":"2024-03-04T16:27:26.510527+0000","last_clean":"2024-03-04T16:27:26.510527+0000","last_became_active":"2024-03-04T16:25:35.903214+0000","last_became_peered":"2024-03-04T16:25:35.903214+0000","last_unstale":"2024-03-04T16:27:26.510527+0000","last_undegraded":"2024-03-04T16:27:26.510527+0000","last_fullsized":"2024-03-04T16:27:26.510527+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:26:47.760650+0000","last_deep_scrub":"286'34","last_deep_scrub_stamp":"2024-03-04T16:16:16.419895+0000","last_clean_scrub_stamp":"2024-03-04T16:26:47.760650+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.335879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":274,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.513162+0000","last_change":"2024-03-04T16:26:25.887125+0000","last_active":"2024-03-04T16:27:26.513162+0000","last_peered":"2024-03-04T16:27:26.513162+0000","last_clean":"2024-03-04T16:27:26.513162+0000","last_became_active":"2024-03-04T16:26:25.884116+0000","last_became_peered":"2024-03-04T16:26:25.884116+0000","last_unstale":"2024-03-04T16:27:26.513162+0000","last_undegraded":"2024-03-04T16:27:26.513162+0000","last_fullsized":"2024-03-04T16:27:26.513162+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.974587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:14:57.996007+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.974587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511843+0000","last_change":"2024-03-04T16:26:31.692570+0000","last_active":"2024-03-04T16:27:26.511843+0000","last_peered":"2024-03-04T16:27:26.511843+0000","last_clean":"2024-03-04T16:27:26.511843+0000","last_became_active":"2024-03-04T16:25:41.966433+0000","last_became_peered":"2024-03-04T16:25:41.966433+0000","last_unstale":"2024-03-04T16:27:26.511843+0000","last_undegraded":"2024-03-04T16:27:26.511843+0000","last_fullsized":"2024-03-04T16:27:26.511843+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:31.692472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.692472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:37.869074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511373+0000","last_change":"2024-03-04T16:26:22.749558+0000","last_active":"2024-03-04T16:27:26.511373+0000","last_peered":"2024-03-04T16:27:26.511373+0000","last_clean":"2024-03-04T16:27:26.511373+0000","last_became_active":"2024-03-04T16:25:41.965271+0000","last_became_peered":"2024-03-04T16:25:41.965271+0000","last_unstale":"2024-03-04T16:27:26.511373+0000","last_undegraded":"2024-03-04T16:27:26.511373+0000","last_fullsized":"2024-03-04T16:27:26.511373+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:22.749468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:22.749468+0000","last_clean_scrub_stamp":"2024-03-04T16:26:22.749468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:49.052669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":730,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512606+0000","last_change":"2024-03-04T16:26:54.787704+0000","last_active":"2024-03-04T16:27:26.512606+0000","last_peered":"2024-03-04T16:27:26.512606+0000","last_clean":"2024-03-04T16:27:26.512606+0000","last_became_active":"2024-03-04T16:26:16.728742+0000","last_became_peered":"2024-03-04T16:26:16.728742+0000","last_unstale":"2024-03-04T16:27:26.512606+0000","last_undegraded":"2024-03-04T16:27:26.512606+0000","last_fullsized":"2024-03-04T16:27:26.512606+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:26:54.787570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.787570+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.943006+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"90'2","reported_seq":1016,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597863+0000","last_change":"2024-03-04T16:26:43.030101+0000","last_active":"2024-03-04T16:27:26.597863+0000","last_peered":"2024-03-04T16:27:26.597863+0000","last_clean":"2024-03-04T16:27:26.597863+0000","last_became_active":"2024-03-04T16:19:04.749822+0000","last_became_peered":"2024-03-04T16:19:04.749822+0000","last_unstale":"2024-03-04T16:27:26.597863+0000","last_undegraded":"2024-03-04T16:27:26.597863+0000","last_fullsized":"2024-03-04T16:27:26.597863+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'2","last_scrub_stamp":"2024-03-04T16:26:43.029998+0000","last_deep_scrub":"90'2","last_deep_scrub_stamp":"2024-03-04T16:25:13.109799+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.029998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.200254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"491'14","reported_seq":343,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598461+0000","last_change":"2024-03-04T16:26:56.789600+0000","last_active":"2024-03-04T16:27:26.598461+0000","last_peered":"2024-03-04T16:27:26.598461+0000","last_clean":"2024-03-04T16:27:26.598461+0000","last_became_active":"2024-03-04T16:25:41.958850+0000","last_became_peered":"2024-03-04T16:25:41.958850+0000","last_unstale":"2024-03-04T16:27:26.598461+0000","last_undegraded":"2024-03-04T16:27:26.598461+0000","last_fullsized":"2024-03-04T16:27:26.598461+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'14","last_scrub_stamp":"2024-03-04T16:26:56.789492+0000","last_deep_scrub":"491'14","last_deep_scrub_stamp":"2024-03-04T16:26:56.789492+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.789492+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:09.530922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"68'1","reported_seq":1064,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596193+0000","last_change":"2024-03-04T16:27:06.794156+0000","last_active":"2024-03-04T16:27:26.596193+0000","last_peered":"2024-03-04T16:27:26.596193+0000","last_clean":"2024-03-04T16:27:26.596193+0000","last_became_active":"2024-03-04T16:25:41.957831+0000","last_became_peered":"2024-03-04T16:25:41.957831+0000","last_unstale":"2024-03-04T16:27:26.596193+0000","last_undegraded":"2024-03-04T16:27:26.596193+0000","last_fullsized":"2024-03-04T16:27:26.596193+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'1","last_scrub_stamp":"2024-03-04T16:27:06.794074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.794074+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:14.275696+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":929,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596904+0000","last_change":"2024-03-04T16:26:24.745956+0000","last_active":"2024-03-04T16:27:26.596904+0000","last_peered":"2024-03-04T16:27:26.596904+0000","last_clean":"2024-03-04T16:27:26.596904+0000","last_became_active":"2024-03-04T16:25:35.916024+0000","last_became_peered":"2024-03-04T16:25:35.916024+0000","last_unstale":"2024-03-04T16:27:26.596904+0000","last_undegraded":"2024-03-04T16:27:26.596904+0000","last_fullsized":"2024-03-04T16:27:26.596904+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:26:24.745848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:24.745848+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:44.037802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1004,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597110+0000","last_change":"2024-03-04T16:26:50.064603+0000","last_active":"2024-03-04T16:27:26.597110+0000","last_peered":"2024-03-04T16:27:26.597110+0000","last_clean":"2024-03-04T16:27:26.597110+0000","last_became_active":"2024-03-04T16:25:41.957834+0000","last_became_peered":"2024-03-04T16:25:41.957834+0000","last_unstale":"2024-03-04T16:27:26.597110+0000","last_undegraded":"2024-03-04T16:27:26.597110+0000","last_fullsized":"2024-03-04T16:27:26.597110+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:50.064451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:21:08.786009+0000","last_clean_scrub_stamp":"2024-03-04T16:26:50.064451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:51.834581+0000","scrub_duration":382,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":527,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596433+0000","last_change":"2024-03-04T16:26:17.697728+0000","last_active":"2024-03-04T16:27:26.596433+0000","last_peered":"2024-03-04T16:27:26.596433+0000","last_clean":"2024-03-04T16:27:26.596433+0000","last_became_active":"2024-03-04T16:19:04.826737+0000","last_became_peered":"2024-03-04T16:19:04.826737+0000","last_unstale":"2024-03-04T16:27:26.596433+0000","last_undegraded":"2024-03-04T16:27:26.596433+0000","last_fullsized":"2024-03-04T16:27:26.596433+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.697632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:17.697632+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.697632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.746709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512319+0000","last_change":"2024-03-04T16:26:40.708684+0000","last_active":"2024-03-04T16:27:26.512319+0000","last_peered":"2024-03-04T16:27:26.512319+0000","last_clean":"2024-03-04T16:27:26.512319+0000","last_became_active":"2024-03-04T16:25:59.530296+0000","last_became_peered":"2024-03-04T16:25:59.530296+0000","last_unstale":"2024-03-04T16:27:26.512319+0000","last_undegraded":"2024-03-04T16:27:26.512319+0000","last_fullsized":"2024-03-04T16:27:26.512319+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.708597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:08.076402+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.708597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.588913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'27","reported_seq":293,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.747890+0000","last_change":"2024-03-04T16:27:26.747928+0000","last_active":"2024-03-04T16:27:26.747890+0000","last_peered":"2024-03-04T16:27:26.747890+0000","last_clean":"2024-03-04T16:27:26.747890+0000","last_became_active":"2024-03-04T16:21:45.434840+0000","last_became_peered":"2024-03-04T16:21:45.434840+0000","last_unstale":"2024-03-04T16:27:26.747890+0000","last_undegraded":"2024-03-04T16:27:26.747890+0000","last_fullsized":"2024-03-04T16:27:26.747890+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:27:26.747811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:26.747811+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.588462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1045,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597193+0000","last_change":"2024-03-04T16:26:40.060557+0000","last_active":"2024-03-04T16:27:26.597193+0000","last_peered":"2024-03-04T16:27:26.597193+0000","last_clean":"2024-03-04T16:27:26.597193+0000","last_became_active":"2024-03-04T16:25:41.959916+0000","last_became_peered":"2024-03-04T16:25:41.959916+0000","last_unstale":"2024-03-04T16:27:26.597193+0000","last_undegraded":"2024-03-04T16:27:26.597193+0000","last_fullsized":"2024-03-04T16:27:26.597193+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.060432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:19.779373+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.060432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.616665+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":267,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596639+0000","last_change":"2024-03-04T16:26:47.800409+0000","last_active":"2024-03-04T16:27:26.596639+0000","last_peered":"2024-03-04T16:27:26.596639+0000","last_clean":"2024-03-04T16:27:26.596639+0000","last_became_active":"2024-03-04T16:25:41.958854+0000","last_became_peered":"2024-03-04T16:25:41.958854+0000","last_unstale":"2024-03-04T16:27:26.596639+0000","last_undegraded":"2024-03-04T16:27:26.596639+0000","last_fullsized":"2024-03-04T16:27:26.596639+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:47.800303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:05:36.943869+0000","last_clean_scrub_stamp":"2024-03-04T16:26:47.800303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.203816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":862,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596590+0000","last_change":"2024-03-04T16:26:23.724658+0000","last_active":"2024-03-04T16:27:26.596590+0000","last_peered":"2024-03-04T16:27:26.596590+0000","last_clean":"2024-03-04T16:27:26.596590+0000","last_became_active":"2024-03-04T16:25:35.903977+0000","last_became_peered":"2024-03-04T16:25:35.903977+0000","last_unstale":"2024-03-04T16:27:26.596590+0000","last_undegraded":"2024-03-04T16:27:26.596590+0000","last_fullsized":"2024-03-04T16:27:26.596590+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:23.724559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:24:11.629031+0000","last_clean_scrub_stamp":"2024-03-04T16:26:23.724559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:40.852625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"458'14","reported_seq":96,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512124+0000","last_change":"2024-03-04T16:27:10.900226+0000","last_active":"2024-03-04T16:27:26.512124+0000","last_peered":"2024-03-04T16:27:26.512124+0000","last_clean":"2024-03-04T16:27:26.512124+0000","last_became_active":"2024-03-04T16:25:35.913075+0000","last_became_peered":"2024-03-04T16:25:35.913075+0000","last_unstale":"2024-03-04T16:27:26.512124+0000","last_undegraded":"2024-03-04T16:27:26.512124+0000","last_fullsized":"2024-03-04T16:27:26.512124+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"458'14","last_scrub_stamp":"2024-03-04T16:27:10.900101+0000","last_deep_scrub":"458'14","last_deep_scrub_stamp":"2024-03-04T16:24:16.155381+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.900101+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:36.184135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512945+0000","last_change":"2024-03-04T16:26:28.901269+0000","last_active":"2024-03-04T16:27:26.512945+0000","last_peered":"2024-03-04T16:27:26.512945+0000","last_clean":"2024-03-04T16:27:26.512945+0000","last_became_active":"2024-03-04T16:25:35.912277+0000","last_became_peered":"2024-03-04T16:25:35.912277+0000","last_unstale":"2024-03-04T16:27:26.512945+0000","last_undegraded":"2024-03-04T16:27:26.512945+0000","last_fullsized":"2024-03-04T16:27:26.512945+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:28.901143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:12:14.287598+0000","last_clean_scrub_stamp":"2024-03-04T16:26:28.901143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:32.682436+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1023,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596959+0000","last_change":"2024-03-04T16:26:25.855926+0000","last_active":"2024-03-04T16:27:26.596959+0000","last_peered":"2024-03-04T16:27:26.596959+0000","last_clean":"2024-03-04T16:27:26.596959+0000","last_became_active":"2024-03-04T16:25:41.951860+0000","last_became_peered":"2024-03-04T16:25:41.951860+0000","last_unstale":"2024-03-04T16:27:26.596959+0000","last_undegraded":"2024-03-04T16:27:26.596959+0000","last_fullsized":"2024-03-04T16:27:26.596959+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:25.855795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:19.018504+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.855795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.161793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1024,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598630+0000","last_change":"2024-03-04T16:26:16.984149+0000","last_active":"2024-03-04T16:27:26.598630+0000","last_peered":"2024-03-04T16:27:26.598630+0000","last_clean":"2024-03-04T16:27:26.598630+0000","last_became_active":"2024-03-04T16:25:59.527423+0000","last_became_peered":"2024-03-04T16:25:59.527423+0000","last_unstale":"2024-03-04T16:27:26.598630+0000","last_undegraded":"2024-03-04T16:27:26.598630+0000","last_fullsized":"2024-03-04T16:27:26.598630+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:16.984069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:16.984069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:33.356645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'27","reported_seq":102,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512407+0000","last_change":"2024-03-04T16:27:19.894464+0000","last_active":"2024-03-04T16:27:26.512407+0000","last_peered":"2024-03-04T16:27:26.512407+0000","last_clean":"2024-03-04T16:27:26.512407+0000","last_became_active":"2024-03-04T16:25:35.909109+0000","last_became_peered":"2024-03-04T16:25:35.909109+0000","last_unstale":"2024-03-04T16:27:26.512407+0000","last_undegraded":"2024-03-04T16:27:26.512407+0000","last_fullsized":"2024-03-04T16:27:26.512407+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-04T16:27:19.894371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:19.894371+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:37.769683+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":862,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:28.908219+0000","last_change":"2024-03-04T16:27:28.908243+0000","last_active":"2024-03-04T16:27:28.908219+0000","last_peered":"2024-03-04T16:27:28.908219+0000","last_clean":"2024-03-04T16:27:28.908219+0000","last_became_active":"2024-03-04T16:26:25.886675+0000","last_became_peered":"2024-03-04T16:26:25.886675+0000","last_unstale":"2024-03-04T16:27:28.908219+0000","last_undegraded":"2024-03-04T16:27:28.908219+0000","last_fullsized":"2024-03-04T16:27:28.908219+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:28.908136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:12.785337+0000","last_clean_scrub_stamp":"2024-03-04T16:27:28.908136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:45.088043+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":594,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512460+0000","last_change":"2024-03-04T16:26:38.680384+0000","last_active":"2024-03-04T16:27:26.512460+0000","last_peered":"2024-03-04T16:27:26.512460+0000","last_clean":"2024-03-04T16:27:26.512460+0000","last_became_active":"2024-03-04T16:26:16.732699+0000","last_became_peered":"2024-03-04T16:26:16.732699+0000","last_unstale":"2024-03-04T16:27:26.512460+0000","last_undegraded":"2024-03-04T16:27:26.512460+0000","last_fullsized":"2024-03-04T16:27:26.512460+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:38.680307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:38.680307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:48.850265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596843+0000","last_change":"2024-03-04T16:26:35.972842+0000","last_active":"2024-03-04T16:27:26.596843+0000","last_peered":"2024-03-04T16:27:26.596843+0000","last_clean":"2024-03-04T16:27:26.596843+0000","last_became_active":"2024-03-04T16:25:41.950167+0000","last_became_peered":"2024-03-04T16:25:41.950167+0000","last_unstale":"2024-03-04T16:27:26.596843+0000","last_undegraded":"2024-03-04T16:27:26.596843+0000","last_fullsized":"2024-03-04T16:27:26.596843+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:35.972703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:35.972703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:47.372306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'25","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596157+0000","last_change":"2024-03-04T16:26:53.747035+0000","last_active":"2024-03-04T16:27:26.596157+0000","last_peered":"2024-03-04T16:27:26.596157+0000","last_clean":"2024-03-04T16:27:26.596157+0000","last_became_active":"2024-03-04T16:25:59.531329+0000","last_became_peered":"2024-03-04T16:25:59.531329+0000","last_unstale":"2024-03-04T16:27:26.596157+0000","last_undegraded":"2024-03-04T16:27:26.596157+0000","last_fullsized":"2024-03-04T16:27:26.596157+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:26:53.746921+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-03-04T16:09:13.633480+0000","last_clean_scrub_stamp":"2024-03-04T16:26:53.746921+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.609935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":414,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512333+0000","last_change":"2024-03-04T16:27:13.954062+0000","last_active":"2024-03-04T16:27:26.512333+0000","last_peered":"2024-03-04T16:27:26.512333+0000","last_clean":"2024-03-04T16:27:26.512333+0000","last_became_active":"2024-03-04T16:26:16.731876+0000","last_became_peered":"2024-03-04T16:26:16.731876+0000","last_unstale":"2024-03-04T16:27:26.512333+0000","last_undegraded":"2024-03-04T16:27:26.512333+0000","last_fullsized":"2024-03-04T16:27:26.512333+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:13.953944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:13.953944+0000","last_clean_scrub_stamp":"2024-03-04T16:27:13.953944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.612571+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1009,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596611+0000","last_change":"2024-03-04T16:26:11.825312+0000","last_active":"2024-03-04T16:27:26.596611+0000","last_peered":"2024-03-04T16:27:26.596611+0000","last_clean":"2024-03-04T16:27:26.596611+0000","last_became_active":"2024-03-04T16:25:35.914323+0000","last_became_peered":"2024-03-04T16:25:35.914323+0000","last_unstale":"2024-03-04T16:27:26.596611+0000","last_undegraded":"2024-03-04T16:27:26.596611+0000","last_fullsized":"2024-03-04T16:27:26.596611+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:11.825229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:11.825229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:30.849743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511698+0000","last_change":"2024-03-04T16:26:19.810601+0000","last_active":"2024-03-04T16:27:26.511698+0000","last_peered":"2024-03-04T16:27:26.511698+0000","last_clean":"2024-03-04T16:27:26.511698+0000","last_became_active":"2024-03-04T16:25:41.960053+0000","last_became_peered":"2024-03-04T16:25:41.960053+0000","last_unstale":"2024-03-04T16:27:26.511698+0000","last_undegraded":"2024-03-04T16:27:26.511698+0000","last_fullsized":"2024-03-04T16:27:26.511698+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.810508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.810508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:46.245891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511753+0000","last_change":"2024-03-04T16:26:21.733482+0000","last_active":"2024-03-04T16:27:26.511753+0000","last_peered":"2024-03-04T16:27:26.511753+0000","last_clean":"2024-03-04T16:27:26.511753+0000","last_became_active":"2024-03-04T16:25:41.958985+0000","last_became_peered":"2024-03-04T16:25:41.958985+0000","last_unstale":"2024-03-04T16:27:26.511753+0000","last_undegraded":"2024-03-04T16:27:26.511753+0000","last_fullsized":"2024-03-04T16:27:26.511753+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:26:21.733338+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-04T16:26:21.733338+0000","last_clean_scrub_stamp":"2024-03-04T16:26:21.733338+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:40.813524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":871,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596526+0000","last_change":"2024-03-04T16:27:06.000477+0000","last_active":"2024-03-04T16:27:26.596526+0000","last_peered":"2024-03-04T16:27:26.596526+0000","last_clean":"2024-03-04T16:27:26.596526+0000","last_became_active":"2024-03-04T16:25:56.057550+0000","last_became_peered":"2024-03-04T16:25:56.057550+0000","last_unstale":"2024-03-04T16:27:26.596526+0000","last_undegraded":"2024-03-04T16:27:26.596526+0000","last_fullsized":"2024-03-04T16:27:26.596526+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:06.000373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:06.000373+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.000373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:09.656090+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"484'11","reported_seq":1328,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598304+0000","last_change":"2024-03-04T16:26:18.033443+0000","last_active":"2024-03-04T16:27:26.598304+0000","last_peered":"2024-03-04T16:27:26.598304+0000","last_clean":"2024-03-04T16:27:26.598304+0000","last_became_active":"2024-03-04T16:25:25.762427+0000","last_became_peered":"2024-03-04T16:25:25.762427+0000","last_unstale":"2024-03-04T16:27:26.598304+0000","last_undegraded":"2024-03-04T16:27:26.598304+0000","last_fullsized":"2024-03-04T16:27:26.598304+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'11","last_scrub_stamp":"2024-03-04T16:26:18.033357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:18.033357+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.480149+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"517'2","reported_seq":1082,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597956+0000","last_change":"2024-03-04T16:27:04.851938+0000","last_active":"2024-03-04T16:27:26.597956+0000","last_peered":"2024-03-04T16:27:26.597956+0000","last_clean":"2024-03-04T16:27:26.597956+0000","last_became_active":"2024-03-04T16:25:41.951420+0000","last_became_peered":"2024-03-04T16:25:41.951420+0000","last_unstale":"2024-03-04T16:27:26.597956+0000","last_undegraded":"2024-03-04T16:27:26.597956+0000","last_fullsized":"2024-03-04T16:27:26.597956+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'2","last_scrub_stamp":"2024-03-04T16:27:04.851837+0000","last_deep_scrub":"517'2","last_deep_scrub_stamp":"2024-03-04T16:27:04.851837+0000","last_clean_scrub_stamp":"2024-03-04T16:27:04.851837+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.303458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'30","reported_seq":637,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512610+0000","last_change":"2024-03-04T16:27:21.893403+0000","last_active":"2024-03-04T16:27:26.512610+0000","last_peered":"2024-03-04T16:27:26.512610+0000","last_clean":"2024-03-04T16:27:26.512610+0000","last_became_active":"2024-03-04T16:26:16.732368+0000","last_became_peered":"2024-03-04T16:26:16.732368+0000","last_unstale":"2024-03-04T16:27:26.512610+0000","last_undegraded":"2024-03-04T16:27:26.512610+0000","last_fullsized":"2024-03-04T16:27:26.512610+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-04T16:27:21.893298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.893298+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.899355+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":87,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512416+0000","last_change":"2024-03-04T16:27:20.014282+0000","last_active":"2024-03-04T16:27:26.512416+0000","last_peered":"2024-03-04T16:27:26.512416+0000","last_clean":"2024-03-04T16:27:26.512416+0000","last_became_active":"2024-03-04T16:25:41.965635+0000","last_became_peered":"2024-03-04T16:25:41.965635+0000","last_unstale":"2024-03-04T16:27:26.512416+0000","last_undegraded":"2024-03-04T16:27:26.512416+0000","last_fullsized":"2024-03-04T16:27:26.512416+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:20.014158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:27:20.014158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:31.341699+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"489'13","reported_seq":109,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.513128+0000","last_change":"2024-03-04T16:26:36.670294+0000","last_active":"2024-03-04T16:27:26.513128+0000","last_peered":"2024-03-04T16:27:26.513128+0000","last_clean":"2024-03-04T16:27:26.513128+0000","last_became_active":"2024-03-04T16:25:35.905277+0000","last_became_peered":"2024-03-04T16:25:35.905277+0000","last_unstale":"2024-03-04T16:27:26.513128+0000","last_undegraded":"2024-03-04T16:27:26.513128+0000","last_fullsized":"2024-03-04T16:27:26.513128+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"489'13","last_scrub_stamp":"2024-03-04T16:26:36.670181+0000","last_deep_scrub":"192'10","last_deep_scrub_stamp":"2024-03-04T16:18:13.365318+0000","last_clean_scrub_stamp":"2024-03-04T16:26:36.670181+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.818094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"505'2","reported_seq":121,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596309+0000","last_change":"2024-03-04T16:26:15.743215+0000","last_active":"2024-03-04T16:27:26.596309+0000","last_peered":"2024-03-04T16:27:26.596309+0000","last_clean":"2024-03-04T16:27:26.596309+0000","last_became_active":"2024-03-04T16:25:56.057518+0000","last_became_peered":"2024-03-04T16:25:56.057518+0000","last_unstale":"2024-03-04T16:27:26.596309+0000","last_undegraded":"2024-03-04T16:27:26.596309+0000","last_fullsized":"2024-03-04T16:27:26.596309+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'2","last_scrub_stamp":"2024-03-04T16:26:15.743125+0000","last_deep_scrub":"505'2","last_deep_scrub_stamp":"2024-03-04T16:26:15.743125+0000","last_clean_scrub_stamp":"2024-03-04T16:26:15.743125+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.073501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"441'21","reported_seq":51,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597889+0000","last_change":"2024-03-04T16:26:58.913422+0000","last_active":"2024-03-04T16:27:26.597889+0000","last_peered":"2024-03-04T16:27:26.597889+0000","last_clean":"2024-03-04T16:27:26.597889+0000","last_became_active":"2024-03-04T16:26:25.882162+0000","last_became_peered":"2024-03-04T16:26:25.882162+0000","last_unstale":"2024-03-04T16:27:26.597889+0000","last_undegraded":"2024-03-04T16:27:26.597889+0000","last_fullsized":"2024-03-04T16:27:26.597889+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:58.913306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:58.913306+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:25.220282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":424,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510636+0000","last_change":"2024-03-04T16:26:27.578848+0000","last_active":"2024-03-04T16:27:26.510636+0000","last_peered":"2024-03-04T16:27:26.510636+0000","last_clean":"2024-03-04T16:27:26.510636+0000","last_became_active":"2024-03-04T16:25:41.957995+0000","last_became_peered":"2024-03-04T16:25:41.957995+0000","last_unstale":"2024-03-04T16:27:26.510636+0000","last_undegraded":"2024-03-04T16:27:26.510636+0000","last_fullsized":"2024-03-04T16:27:26.510636+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:27.578727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:27.578727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.415089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":82,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.513233+0000","last_change":"2024-03-04T16:26:13.739831+0000","last_active":"2024-03-04T16:27:26.513233+0000","last_peered":"2024-03-04T16:27:26.513233+0000","last_clean":"2024-03-04T16:27:26.513233+0000","last_became_active":"2024-03-04T16:25:35.913274+0000","last_became_peered":"2024-03-04T16:25:35.913274+0000","last_unstale":"2024-03-04T16:27:26.513233+0000","last_undegraded":"2024-03-04T16:27:26.513233+0000","last_fullsized":"2024-03-04T16:27:26.513233+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:13.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:13.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596903+0000","last_change":"2024-03-04T16:27:15.882985+0000","last_active":"2024-03-04T16:27:26.596903+0000","last_peered":"2024-03-04T16:27:26.596903+0000","last_clean":"2024-03-04T16:27:26.596903+0000","last_became_active":"2024-03-04T16:25:41.950697+0000","last_became_peered":"2024-03-04T16:25:41.950697+0000","last_unstale":"2024-03-04T16:27:26.596903+0000","last_undegraded":"2024-03-04T16:27:26.596903+0000","last_fullsized":"2024-03-04T16:27:26.596903+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:15.882861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:59.857747+0000","last_clean_scrub_stamp":"2024-03-04T16:27:15.882861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:29.402354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'21","reported_seq":910,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597010+0000","last_change":"2024-03-04T16:26:48.870663+0000","last_active":"2024-03-04T16:27:26.597010+0000","last_peered":"2024-03-04T16:27:26.597010+0000","last_clean":"2024-03-04T16:27:26.597010+0000","last_became_active":"2024-03-04T16:25:35.907039+0000","last_became_peered":"2024-03-04T16:25:35.907039+0000","last_unstale":"2024-03-04T16:27:26.597010+0000","last_undegraded":"2024-03-04T16:27:26.597010+0000","last_fullsized":"2024-03-04T16:27:26.597010+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:26:48.870514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.870514+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.628672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":105,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596855+0000","last_change":"2024-03-04T16:27:05.929412+0000","last_active":"2024-03-04T16:27:26.596855+0000","last_peered":"2024-03-04T16:27:26.596855+0000","last_clean":"2024-03-04T16:27:26.596855+0000","last_became_active":"2024-03-04T16:25:41.958001+0000","last_became_peered":"2024-03-04T16:25:41.958001+0000","last_unstale":"2024-03-04T16:27:26.596855+0000","last_undegraded":"2024-03-04T16:27:26.596855+0000","last_fullsized":"2024-03-04T16:27:26.596855+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:05.929284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:15:11.216415+0000","last_clean_scrub_stamp":"2024-03-04T16:27:05.929284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.779254+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598513+0000","last_change":"2024-03-04T16:27:09.067065+0000","last_active":"2024-03-04T16:27:26.598513+0000","last_peered":"2024-03-04T16:27:26.598513+0000","last_clean":"2024-03-04T16:27:26.598513+0000","last_became_active":"2024-03-04T16:25:35.915465+0000","last_became_peered":"2024-03-04T16:25:35.915465+0000","last_unstale":"2024-03-04T16:27:26.598513+0000","last_undegraded":"2024-03-04T16:27:26.598513+0000","last_fullsized":"2024-03-04T16:27:26.598513+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:09.066955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:09:15.043855+0000","last_clean_scrub_stamp":"2024-03-04T16:27:09.066955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:25.935375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":279,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596446+0000","last_change":"2024-03-04T16:27:00.672881+0000","last_active":"2024-03-04T16:27:26.596446+0000","last_peered":"2024-03-04T16:27:26.596446+0000","last_clean":"2024-03-04T16:27:26.596446+0000","last_became_active":"2024-03-04T16:26:25.884905+0000","last_became_peered":"2024-03-04T16:26:25.884905+0000","last_unstale":"2024-03-04T16:27:26.596446+0000","last_undegraded":"2024-03-04T16:27:26.596446+0000","last_fullsized":"2024-03-04T16:27:26.596446+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:00.672777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:03:46.718188+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.672777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:05.739226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'21","reported_seq":98,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.513254+0000","last_change":"2024-03-04T16:27:00.813329+0000","last_active":"2024-03-04T16:27:26.513254+0000","last_peered":"2024-03-04T16:27:26.513254+0000","last_clean":"2024-03-04T16:27:26.513254+0000","last_became_active":"2024-03-04T16:25:35.912617+0000","last_became_peered":"2024-03-04T16:25:35.912617+0000","last_unstale":"2024-03-04T16:27:26.513254+0000","last_undegraded":"2024-03-04T16:27:26.513254+0000","last_fullsized":"2024-03-04T16:27:26.513254+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-04T16:27:00.813233+0000","last_deep_scrub":"86'9","last_deep_scrub_stamp":"2024-03-04T16:03:52.421285+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.813233+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:26.664975+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1013,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597243+0000","last_change":"2024-03-04T16:26:30.938999+0000","last_active":"2024-03-04T16:27:26.597243+0000","last_peered":"2024-03-04T16:27:26.597243+0000","last_clean":"2024-03-04T16:27:26.597243+0000","last_became_active":"2024-03-04T16:21:45.424279+0000","last_became_peered":"2024-03-04T16:21:45.424279+0000","last_unstale":"2024-03-04T16:27:26.597243+0000","last_undegraded":"2024-03-04T16:27:26.597243+0000","last_fullsized":"2024-03-04T16:27:26.597243+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:30.938891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.938891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.799044+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"68'2","reported_seq":1087,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597530+0000","last_change":"2024-03-04T16:27:17.895718+0000","last_active":"2024-03-04T16:27:26.597530+0000","last_peered":"2024-03-04T16:27:26.597530+0000","last_clean":"2024-03-04T16:27:26.597530+0000","last_became_active":"2024-03-04T16:25:41.951605+0000","last_became_peered":"2024-03-04T16:25:41.951605+0000","last_unstale":"2024-03-04T16:27:26.597530+0000","last_undegraded":"2024-03-04T16:27:26.597530+0000","last_fullsized":"2024-03-04T16:27:26.597530+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-04T16:27:17.895612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.895612+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.319967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":57,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596516+0000","last_change":"2024-03-04T16:27:10.069698+0000","last_active":"2024-03-04T16:27:26.596516+0000","last_peered":"2024-03-04T16:27:26.596516+0000","last_clean":"2024-03-04T16:27:26.596516+0000","last_became_active":"2024-03-04T16:26:25.877388+0000","last_became_peered":"2024-03-04T16:26:25.877388+0000","last_unstale":"2024-03-04T16:27:26.596516+0000","last_undegraded":"2024-03-04T16:27:26.596516+0000","last_fullsized":"2024-03-04T16:27:26.596516+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-04T16:27:10.069583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:10.069583+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:21.478046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"492'15","reported_seq":1374,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597223+0000","last_change":"2024-03-04T16:27:21.827455+0000","last_active":"2024-03-04T16:27:26.597223+0000","last_peered":"2024-03-04T16:27:26.597223+0000","last_clean":"2024-03-04T16:27:26.597223+0000","last_became_active":"2024-03-04T16:25:41.952470+0000","last_became_peered":"2024-03-04T16:25:41.952470+0000","last_unstale":"2024-03-04T16:27:26.597223+0000","last_undegraded":"2024-03-04T16:27:26.597223+0000","last_fullsized":"2024-03-04T16:27:26.597223+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"492'15","last_scrub_stamp":"2024-03-04T16:27:21.827308+0000","last_deep_scrub":"492'15","last_deep_scrub_stamp":"2024-03-04T16:25:52.919379+0000","last_clean_scrub_stamp":"2024-03-04T16:27:21.827308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.014368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512781+0000","last_change":"2024-03-04T16:27:09.184891+0000","last_active":"2024-03-04T16:27:26.512781+0000","last_peered":"2024-03-04T16:27:26.512781+0000","last_clean":"2024-03-04T16:27:26.512781+0000","last_became_active":"2024-03-04T16:25:35.912995+0000","last_became_peered":"2024-03-04T16:25:35.912995+0000","last_unstale":"2024-03-04T16:27:26.512781+0000","last_undegraded":"2024-03-04T16:27:26.512781+0000","last_fullsized":"2024-03-04T16:27:26.512781+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:09.184789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:52.766115+0000","last_clean_scrub_stamp":"2024-03-04T16:27:09.184789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.784865+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1027,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597834+0000","last_change":"2024-03-04T16:26:45.021081+0000","last_active":"2024-03-04T16:27:26.597834+0000","last_peered":"2024-03-04T16:27:26.597834+0000","last_clean":"2024-03-04T16:27:26.597834+0000","last_became_active":"2024-03-04T16:25:41.952002+0000","last_became_peered":"2024-03-04T16:25:41.952002+0000","last_unstale":"2024-03-04T16:27:26.597834+0000","last_undegraded":"2024-03-04T16:27:26.597834+0000","last_fullsized":"2024-03-04T16:27:26.597834+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:45.020970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:16.071086+0000","last_clean_scrub_stamp":"2024-03-04T16:26:45.020970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:02.566679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":116,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511346+0000","last_change":"2024-03-04T16:27:04.776193+0000","last_active":"2024-03-04T16:27:26.511346+0000","last_peered":"2024-03-04T16:27:26.511346+0000","last_clean":"2024-03-04T16:27:26.511346+0000","last_became_active":"2024-03-04T16:26:25.884201+0000","last_became_peered":"2024-03-04T16:26:25.884201+0000","last_unstale":"2024-03-04T16:27:26.511346+0000","last_undegraded":"2024-03-04T16:27:26.511346+0000","last_fullsized":"2024-03-04T16:27:26.511346+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-04T16:27:04.776063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:04.776063+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.513891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":418,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510095+0000","last_change":"2024-03-04T16:26:52.787335+0000","last_active":"2024-03-04T16:27:26.510095+0000","last_peered":"2024-03-04T16:27:26.510095+0000","last_clean":"2024-03-04T16:27:26.510095+0000","last_became_active":"2024-03-04T16:24:49.071183+0000","last_became_peered":"2024-03-04T16:24:49.071183+0000","last_unstale":"2024-03-04T16:27:26.510095+0000","last_undegraded":"2024-03-04T16:27:26.510095+0000","last_fullsized":"2024-03-04T16:27:26.510095+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:52.787244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.787244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:16.662941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597157+0000","last_change":"2024-03-04T16:26:17.877692+0000","last_active":"2024-03-04T16:27:26.597157+0000","last_peered":"2024-03-04T16:27:26.597157+0000","last_clean":"2024-03-04T16:27:26.597157+0000","last_became_active":"2024-03-04T16:25:35.873601+0000","last_became_peered":"2024-03-04T16:25:35.873601+0000","last_unstale":"2024-03-04T16:27:26.597157+0000","last_undegraded":"2024-03-04T16:27:26.597157+0000","last_fullsized":"2024-03-04T16:27:26.597157+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:17.877611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:17.877611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:28.351289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":424,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510361+0000","last_change":"2024-03-04T16:26:26.529455+0000","last_active":"2024-03-04T16:27:26.510361+0000","last_peered":"2024-03-04T16:27:26.510361+0000","last_clean":"2024-03-04T16:27:26.510361+0000","last_became_active":"2024-03-04T16:26:25.884184+0000","last_became_peered":"2024-03-04T16:26:25.884184+0000","last_unstale":"2024-03-04T16:27:26.510361+0000","last_undegraded":"2024-03-04T16:27:26.510361+0000","last_fullsized":"2024-03-04T16:27:26.510361+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.529374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:15.548884+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.529374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.510339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597466+0000","last_change":"2024-03-04T16:26:26.875537+0000","last_active":"2024-03-04T16:27:26.597466+0000","last_peered":"2024-03-04T16:27:26.597466+0000","last_clean":"2024-03-04T16:27:26.597466+0000","last_became_active":"2024-03-04T16:26:25.877942+0000","last_became_peered":"2024-03-04T16:26:25.877942+0000","last_unstale":"2024-03-04T16:27:26.597466+0000","last_undegraded":"2024-03-04T16:27:26.597466+0000","last_fullsized":"2024-03-04T16:27:26.597466+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:26.875453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:26.875453+0000","last_clean_scrub_stamp":"2024-03-04T16:26:26.875453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:31.978054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'52","reported_seq":339,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597674+0000","last_change":"2024-03-04T16:27:06.789992+0000","last_active":"2024-03-04T16:27:26.597674+0000","last_peered":"2024-03-04T16:27:26.597674+0000","last_clean":"2024-03-04T16:27:26.597674+0000","last_became_active":"2024-03-04T16:22:24.826414+0000","last_became_peered":"2024-03-04T16:22:24.826414+0000","last_unstale":"2024-03-04T16:27:26.597674+0000","last_undegraded":"2024-03-04T16:27:26.597674+0000","last_fullsized":"2024-03-04T16:27:26.597674+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'52","last_scrub_stamp":"2024-03-04T16:27:06.789852+0000","last_deep_scrub":"248'36","last_deep_scrub_stamp":"2024-03-04T16:13:13.646813+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.789852+0000","objects_scrubbed":9,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:26.566175+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":559,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.595963+0000","last_change":"2024-03-04T16:26:43.035072+0000","last_active":"2024-03-04T16:27:26.595963+0000","last_peered":"2024-03-04T16:27:26.595963+0000","last_clean":"2024-03-04T16:27:26.595963+0000","last_became_active":"2024-03-04T16:25:56.057201+0000","last_became_peered":"2024-03-04T16:25:56.057201+0000","last_unstale":"2024-03-04T16:27:26.595963+0000","last_undegraded":"2024-03-04T16:27:26.595963+0000","last_fullsized":"2024-03-04T16:27:26.595963+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:43.034972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:04.117139+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.034972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.501653+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":450,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510345+0000","last_change":"2024-03-04T16:27:18.888603+0000","last_active":"2024-03-04T16:27:26.510345+0000","last_peered":"2024-03-04T16:27:26.510345+0000","last_clean":"2024-03-04T16:27:26.510345+0000","last_became_active":"2024-03-04T16:25:41.957993+0000","last_became_peered":"2024-03-04T16:25:41.957993+0000","last_unstale":"2024-03-04T16:27:26.510345+0000","last_undegraded":"2024-03-04T16:27:26.510345+0000","last_fullsized":"2024-03-04T16:27:26.510345+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:18.888522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:08:20.664848+0000","last_clean_scrub_stamp":"2024-03-04T16:27:18.888522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:30.252697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1030,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597499+0000","last_change":"2024-03-04T16:26:49.939876+0000","last_active":"2024-03-04T16:27:26.597499+0000","last_peered":"2024-03-04T16:27:26.597499+0000","last_clean":"2024-03-04T16:27:26.597499+0000","last_became_active":"2024-03-04T16:25:35.914551+0000","last_became_peered":"2024-03-04T16:25:35.914551+0000","last_unstale":"2024-03-04T16:27:26.597499+0000","last_undegraded":"2024-03-04T16:27:26.597499+0000","last_fullsized":"2024-03-04T16:27:26.597499+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:49.939785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:44.833507+0000","last_clean_scrub_stamp":"2024-03-04T16:26:49.939785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:00.738998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"441'25","reported_seq":910,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596550+0000","last_change":"2024-03-04T16:26:04.596514+0000","last_active":"2024-03-04T16:27:26.596550+0000","last_peered":"2024-03-04T16:27:26.596550+0000","last_clean":"2024-03-04T16:27:26.596550+0000","last_became_active":"2024-03-04T16:21:10.564447+0000","last_became_peered":"2024-03-04T16:21:10.564447+0000","last_unstale":"2024-03-04T16:27:26.596550+0000","last_undegraded":"2024-03-04T16:27:26.596550+0000","last_fullsized":"2024-03-04T16:27:26.596550+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-04T16:26:04.596406+0000","last_deep_scrub":"240'19","last_deep_scrub_stamp":"2024-03-04T16:13:16.698648+0000","last_clean_scrub_stamp":"2024-03-04T16:26:04.596406+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:32.761300+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":88,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512857+0000","last_change":"2024-03-04T16:26:39.736912+0000","last_active":"2024-03-04T16:27:26.512857+0000","last_peered":"2024-03-04T16:27:26.512857+0000","last_clean":"2024-03-04T16:27:26.512857+0000","last_became_active":"2024-03-04T16:25:41.945618+0000","last_became_peered":"2024-03-04T16:25:41.945618+0000","last_unstale":"2024-03-04T16:27:26.512857+0000","last_undegraded":"2024-03-04T16:27:26.512857+0000","last_fullsized":"2024-03-04T16:27:26.512857+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:39.736804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:39.736804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.849666+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":1038,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596889+0000","last_change":"2024-03-04T16:27:18.099142+0000","last_active":"2024-03-04T16:27:26.596889+0000","last_peered":"2024-03-04T16:27:26.596889+0000","last_clean":"2024-03-04T16:27:26.596889+0000","last_became_active":"2024-03-04T16:21:45.425232+0000","last_became_peered":"2024-03-04T16:21:45.425232+0000","last_unstale":"2024-03-04T16:27:26.596889+0000","last_undegraded":"2024-03-04T16:27:26.596889+0000","last_fullsized":"2024-03-04T16:27:26.596889+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-04T16:27:18.099030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:41.630915+0000","last_clean_scrub_stamp":"2024-03-04T16:27:18.099030+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:30.139552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00055153600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511533+0000","last_change":"2024-03-04T16:26:43.800073+0000","last_active":"2024-03-04T16:27:26.511533+0000","last_peered":"2024-03-04T16:27:26.511533+0000","last_clean":"2024-03-04T16:27:26.511533+0000","last_became_active":"2024-03-04T16:25:41.959686+0000","last_became_peered":"2024-03-04T16:25:41.959686+0000","last_unstale":"2024-03-04T16:27:26.511533+0000","last_undegraded":"2024-03-04T16:27:26.511533+0000","last_fullsized":"2024-03-04T16:27:26.511533+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:43.799888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:18:08.651791+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.799888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:44.665976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"496'59","reported_seq":109,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596458+0000","last_change":"2024-03-04T16:26:43.864588+0000","last_active":"2024-03-04T16:27:26.596458+0000","last_peered":"2024-03-04T16:27:26.596458+0000","last_clean":"2024-03-04T16:27:26.596458+0000","last_became_active":"2024-03-04T16:25:59.530455+0000","last_became_peered":"2024-03-04T16:25:59.530455+0000","last_unstale":"2024-03-04T16:27:26.596458+0000","last_undegraded":"2024-03-04T16:27:26.596458+0000","last_fullsized":"2024-03-04T16:27:26.596458+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'59","last_scrub_stamp":"2024-03-04T16:26:43.864484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:43.864484+0000","objects_scrubbed":10,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.976346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":2148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":869,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596743+0000","last_change":"2024-03-04T16:26:54.810925+0000","last_active":"2024-03-04T16:27:26.596743+0000","last_peered":"2024-03-04T16:27:26.596743+0000","last_clean":"2024-03-04T16:27:26.596743+0000","last_became_active":"2024-03-04T16:25:41.954560+0000","last_became_peered":"2024-03-04T16:25:41.954560+0000","last_unstale":"2024-03-04T16:27:26.596743+0000","last_undegraded":"2024-03-04T16:27:26.596743+0000","last_fullsized":"2024-03-04T16:27:26.596743+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:54.810827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:54.810827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:01.618062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597581+0000","last_change":"2024-03-04T16:26:19.981556+0000","last_active":"2024-03-04T16:27:26.597581+0000","last_peered":"2024-03-04T16:27:26.597581+0000","last_clean":"2024-03-04T16:27:26.597581+0000","last_became_active":"2024-03-04T16:25:35.905161+0000","last_became_peered":"2024-03-04T16:25:35.905161+0000","last_unstale":"2024-03-04T16:27:26.597581+0000","last_undegraded":"2024-03-04T16:27:26.597581+0000","last_fullsized":"2024-03-04T16:27:26.597581+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:19.981459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.981459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:27.541909+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1031,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596192+0000","last_change":"2024-03-04T16:26:34.783238+0000","last_active":"2024-03-04T16:27:26.596192+0000","last_peered":"2024-03-04T16:27:26.596192+0000","last_clean":"2024-03-04T16:27:26.596192+0000","last_became_active":"2024-03-04T16:21:45.437084+0000","last_became_peered":"2024-03-04T16:21:45.437084+0000","last_unstale":"2024-03-04T16:27:26.596192+0000","last_undegraded":"2024-03-04T16:27:26.596192+0000","last_fullsized":"2024-03-04T16:27:26.596192+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:34.783156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:34.783156+0000","last_clean_scrub_stamp":"2024-03-04T16:26:34.783156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:35.437461+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":433,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510208+0000","last_change":"2024-03-04T16:26:41.645338+0000","last_active":"2024-03-04T16:27:26.510208+0000","last_peered":"2024-03-04T16:27:26.510208+0000","last_clean":"2024-03-04T16:27:26.510208+0000","last_became_active":"2024-03-04T16:25:35.876577+0000","last_became_peered":"2024-03-04T16:25:35.876577+0000","last_unstale":"2024-03-04T16:27:26.510208+0000","last_undegraded":"2024-03-04T16:27:26.510208+0000","last_fullsized":"2024-03-04T16:27:26.510208+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:41.645255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:16:10.021439+0000","last_clean_scrub_stamp":"2024-03-04T16:26:41.645255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:04.793334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":80,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596980+0000","last_change":"2024-03-04T16:26:30.847294+0000","last_active":"2024-03-04T16:27:26.596980+0000","last_peered":"2024-03-04T16:27:26.596980+0000","last_clean":"2024-03-04T16:27:26.596980+0000","last_became_active":"2024-03-04T16:25:59.530003+0000","last_became_peered":"2024-03-04T16:25:59.530003+0000","last_unstale":"2024-03-04T16:27:26.596980+0000","last_undegraded":"2024-03-04T16:27:26.596980+0000","last_fullsized":"2024-03-04T16:27:26.596980+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-04T16:26:30.847177+0000","last_deep_scrub":"441'26","last_deep_scrub_stamp":"2024-03-04T16:26:30.847177+0000","last_clean_scrub_stamp":"2024-03-04T16:26:30.847177+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.800645+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":412,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510646+0000","last_change":"2024-03-04T16:26:40.598225+0000","last_active":"2024-03-04T16:27:26.510646+0000","last_peered":"2024-03-04T16:27:26.510646+0000","last_clean":"2024-03-04T16:27:26.510646+0000","last_became_active":"2024-03-04T16:19:04.756025+0000","last_became_peered":"2024-03-04T16:19:04.756025+0000","last_unstale":"2024-03-04T16:27:26.510646+0000","last_undegraded":"2024-03-04T16:27:26.510646+0000","last_fullsized":"2024-03-04T16:27:26.510646+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:40.598141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:40.598141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:47.374222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1025,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598378+0000","last_change":"2024-03-04T16:26:52.070794+0000","last_active":"2024-03-04T16:27:26.598378+0000","last_peered":"2024-03-04T16:27:26.598378+0000","last_clean":"2024-03-04T16:27:26.598378+0000","last_became_active":"2024-03-04T16:25:41.968664+0000","last_became_peered":"2024-03-04T16:25:41.968664+0000","last_unstale":"2024-03-04T16:27:26.598378+0000","last_undegraded":"2024-03-04T16:27:26.598378+0000","last_fullsized":"2024-03-04T16:27:26.598378+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:52.070677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.070677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.891827+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"80'130","reported_seq":1331,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596647+0000","last_change":"2024-03-04T16:26:31.917342+0000","last_active":"2024-03-04T16:27:26.596647+0000","last_peered":"2024-03-04T16:27:26.596647+0000","last_clean":"2024-03-04T16:27:26.596647+0000","last_became_active":"2024-03-04T16:21:48.434375+0000","last_became_peered":"2024-03-04T16:21:48.434375+0000","last_unstale":"2024-03-04T16:27:26.596647+0000","last_undegraded":"2024-03-04T16:27:26.596647+0000","last_fullsized":"2024-03-04T16:27:26.596647+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'130","last_scrub_stamp":"2024-03-04T16:26:31.917158+0000","last_deep_scrub":"77'119","last_deep_scrub_stamp":"2024-03-04T16:03:12.851677+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.917158+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:39.727777+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":910,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512262+0000","last_change":"2024-03-04T16:27:14.879085+0000","last_active":"2024-03-04T16:27:26.512262+0000","last_peered":"2024-03-04T16:27:26.512262+0000","last_clean":"2024-03-04T16:27:26.512262+0000","last_became_active":"2024-03-04T16:26:16.731937+0000","last_became_peered":"2024-03-04T16:26:16.731937+0000","last_unstale":"2024-03-04T16:27:26.512262+0000","last_undegraded":"2024-03-04T16:27:26.512262+0000","last_fullsized":"2024-03-04T16:27:26.512262+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:14.878977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:27:14.878977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:41.023143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"491'52","reported_seq":389,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598735+0000","last_change":"2024-03-04T16:26:45.826140+0000","last_active":"2024-03-04T16:27:26.598735+0000","last_peered":"2024-03-04T16:27:26.598735+0000","last_clean":"2024-03-04T16:27:26.598735+0000","last_became_active":"2024-03-04T16:25:41.972439+0000","last_became_peered":"2024-03-04T16:25:41.972439+0000","last_unstale":"2024-03-04T16:27:26.598735+0000","last_undegraded":"2024-03-04T16:27:26.598735+0000","last_fullsized":"2024-03-04T16:27:26.598735+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'52","last_scrub_stamp":"2024-03-04T16:26:45.825987+0000","last_deep_scrub":"391'38","last_deep_scrub_stamp":"2024-03-04T16:21:02.538605+0000","last_clean_scrub_stamp":"2024-03-04T16:26:45.825987+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:46.472060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597985+0000","last_change":"2024-03-04T16:27:23.891109+0000","last_active":"2024-03-04T16:27:26.597985+0000","last_peered":"2024-03-04T16:27:26.597985+0000","last_clean":"2024-03-04T16:27:26.597985+0000","last_became_active":"2024-03-04T16:19:04.750402+0000","last_became_peered":"2024-03-04T16:19:04.750402+0000","last_unstale":"2024-03-04T16:27:26.597985+0000","last_undegraded":"2024-03-04T16:27:26.597985+0000","last_fullsized":"2024-03-04T16:27:26.597985+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:23.891001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:27:23.891001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:53.310100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":68,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596510+0000","last_change":"2024-03-04T16:26:32.781033+0000","last_active":"2024-03-04T16:27:26.596510+0000","last_peered":"2024-03-04T16:27:26.596510+0000","last_clean":"2024-03-04T16:27:26.596510+0000","last_became_active":"2024-03-04T16:25:59.531011+0000","last_became_peered":"2024-03-04T16:25:59.531011+0000","last_unstale":"2024-03-04T16:27:26.596510+0000","last_undegraded":"2024-03-04T16:27:26.596510+0000","last_fullsized":"2024-03-04T16:27:26.596510+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:32.780879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:12.831863+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.780879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:34.448886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":883,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596898+0000","last_change":"2024-03-04T16:26:31.776684+0000","last_active":"2024-03-04T16:27:26.596898+0000","last_peered":"2024-03-04T16:27:26.596898+0000","last_clean":"2024-03-04T16:27:26.596898+0000","last_became_active":"2024-03-04T16:25:41.955219+0000","last_became_peered":"2024-03-04T16:25:41.955219+0000","last_unstale":"2024-03-04T16:27:26.596898+0000","last_undegraded":"2024-03-04T16:27:26.596898+0000","last_fullsized":"2024-03-04T16:27:26.596898+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:31.776609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:31.776609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:34.783122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"172'6","reported_seq":2543,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.888568+0000","last_change":"2024-03-04T16:27:26.888568+0000","last_active":"2024-03-04T16:27:26.888568+0000","last_peered":"2024-03-04T16:27:26.888568+0000","last_clean":"2024-03-04T16:27:26.888568+0000","last_became_active":"2024-03-04T16:25:59.522981+0000","last_became_peered":"2024-03-04T16:25:59.522981+0000","last_unstale":"2024-03-04T16:27:26.888568+0000","last_undegraded":"2024-03-04T16:27:26.888568+0000","last_fullsized":"2024-03-04T16:27:26.888568+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'6","last_scrub_stamp":"2024-03-04T16:27:26.888456+0000","last_deep_scrub":"172'6","last_deep_scrub_stamp":"2024-03-04T16:09:21.827277+0000","last_clean_scrub_stamp":"2024-03-04T16:27:26.888456+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:44.986481+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0031712189999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1471,"num_read_kb":1466,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":961,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596713+0000","last_change":"2024-03-04T16:27:08.865537+0000","last_active":"2024-03-04T16:27:26.596713+0000","last_peered":"2024-03-04T16:27:26.596713+0000","last_clean":"2024-03-04T16:27:26.596713+0000","last_became_active":"2024-03-04T16:22:24.838293+0000","last_became_peered":"2024-03-04T16:22:24.838293+0000","last_unstale":"2024-03-04T16:27:26.596713+0000","last_undegraded":"2024-03-04T16:27:26.596713+0000","last_fullsized":"2024-03-04T16:27:26.596713+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-04T16:27:08.865437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.865437+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.052042+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598179+0000","last_change":"2024-03-04T16:26:25.988336+0000","last_active":"2024-03-04T16:27:26.598179+0000","last_peered":"2024-03-04T16:27:26.598179+0000","last_clean":"2024-03-04T16:27:26.598179+0000","last_became_active":"2024-03-04T16:26:25.884588+0000","last_became_peered":"2024-03-04T16:26:25.884588+0000","last_unstale":"2024-03-04T16:27:26.598179+0000","last_undegraded":"2024-03-04T16:27:26.598179+0000","last_fullsized":"2024-03-04T16:27:26.598179+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:25.988227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:25.988227+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.988227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:49.637237+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1026,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596040+0000","last_change":"2024-03-04T16:26:46.757066+0000","last_active":"2024-03-04T16:27:26.596040+0000","last_peered":"2024-03-04T16:27:26.596040+0000","last_clean":"2024-03-04T16:27:26.596040+0000","last_became_active":"2024-03-04T16:21:45.442233+0000","last_became_peered":"2024-03-04T16:21:45.442233+0000","last_unstale":"2024-03-04T16:27:26.596040+0000","last_undegraded":"2024-03-04T16:27:26.596040+0000","last_fullsized":"2024-03-04T16:27:26.596040+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:46.756942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:35.788869+0000","last_clean_scrub_stamp":"2024-03-04T16:26:46.756942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:51.824085+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596670+0000","last_change":"2024-03-04T16:26:06.823971+0000","last_active":"2024-03-04T16:27:26.596670+0000","last_peered":"2024-03-04T16:27:26.596670+0000","last_clean":"2024-03-04T16:27:26.596670+0000","last_became_active":"2024-03-04T16:19:04.769208+0000","last_became_peered":"2024-03-04T16:19:04.769208+0000","last_unstale":"2024-03-04T16:27:26.596670+0000","last_undegraded":"2024-03-04T16:27:26.596670+0000","last_fullsized":"2024-03-04T16:27:26.596670+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:06.823861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:06.823861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'47","reported_seq":890,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596308+0000","last_change":"2024-03-04T16:26:32.782094+0000","last_active":"2024-03-04T16:27:26.596308+0000","last_peered":"2024-03-04T16:27:26.596308+0000","last_clean":"2024-03-04T16:27:26.596308+0000","last_became_active":"2024-03-04T16:25:56.057569+0000","last_became_peered":"2024-03-04T16:25:56.057569+0000","last_unstale":"2024-03-04T16:27:26.596308+0000","last_undegraded":"2024-03-04T16:27:26.596308+0000","last_fullsized":"2024-03-04T16:27:26.596308+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'47","last_scrub_stamp":"2024-03-04T16:26:32.782005+0000","last_deep_scrub":"159'27","last_deep_scrub_stamp":"2024-03-04T16:08:24.229746+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.782005+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.396594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597522+0000","last_change":"2024-03-04T16:27:17.611404+0000","last_active":"2024-03-04T16:27:26.597522+0000","last_peered":"2024-03-04T16:27:26.597522+0000","last_clean":"2024-03-04T16:27:26.597522+0000","last_became_active":"2024-03-04T16:21:45.412152+0000","last_became_peered":"2024-03-04T16:21:45.412152+0000","last_unstale":"2024-03-04T16:27:26.597522+0000","last_undegraded":"2024-03-04T16:27:26.597522+0000","last_fullsized":"2024-03-04T16:27:26.597522+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:17.611297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:22:08.770293+0000","last_clean_scrub_stamp":"2024-03-04T16:27:17.611297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.488811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1034,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596014+0000","last_change":"2024-03-04T16:26:02.724516+0000","last_active":"2024-03-04T16:27:26.596014+0000","last_peered":"2024-03-04T16:27:26.596014+0000","last_clean":"2024-03-04T16:27:26.596014+0000","last_became_active":"2024-03-04T16:25:35.905478+0000","last_became_peered":"2024-03-04T16:25:35.905478+0000","last_unstale":"2024-03-04T16:27:26.596014+0000","last_undegraded":"2024-03-04T16:27:26.596014+0000","last_fullsized":"2024-03-04T16:27:26.596014+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:02.724383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:02.724383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:28.900944+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1037,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598335+0000","last_change":"2024-03-04T16:27:08.113585+0000","last_active":"2024-03-04T16:27:26.598335+0000","last_peered":"2024-03-04T16:27:26.598335+0000","last_clean":"2024-03-04T16:27:26.598335+0000","last_became_active":"2024-03-04T16:25:35.915319+0000","last_became_peered":"2024-03-04T16:25:35.915319+0000","last_unstale":"2024-03-04T16:27:26.598335+0000","last_undegraded":"2024-03-04T16:27:26.598335+0000","last_fullsized":"2024-03-04T16:27:26.598335+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:08.113444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:20:51.445890+0000","last_clean_scrub_stamp":"2024-03-04T16:27:08.113444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:22.295481+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":98,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596512+0000","last_change":"2024-03-04T16:26:33.773431+0000","last_active":"2024-03-04T16:27:26.596512+0000","last_peered":"2024-03-04T16:27:26.596512+0000","last_clean":"2024-03-04T16:27:26.596512+0000","last_became_active":"2024-03-04T16:26:25.883379+0000","last_became_peered":"2024-03-04T16:26:25.883379+0000","last_unstale":"2024-03-04T16:27:26.596512+0000","last_undegraded":"2024-03-04T16:27:26.596512+0000","last_fullsized":"2024-03-04T16:27:26.596512+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:26:33.773330+0000","last_deep_scrub":"240'25","last_deep_scrub_stamp":"2024-03-04T16:13:16.605307+0000","last_clean_scrub_stamp":"2024-03-04T16:26:33.773330+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:50.397293+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"494'13","reported_seq":164,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596243+0000","last_change":"2024-03-04T16:26:19.742627+0000","last_active":"2024-03-04T16:27:26.596243+0000","last_peered":"2024-03-04T16:27:26.596243+0000","last_clean":"2024-03-04T16:27:26.596243+0000","last_became_active":"2024-03-04T16:25:56.057430+0000","last_became_peered":"2024-03-04T16:25:56.057430+0000","last_unstale":"2024-03-04T16:27:26.596243+0000","last_undegraded":"2024-03-04T16:27:26.596243+0000","last_fullsized":"2024-03-04T16:27:26.596243+0000","mapping_epoch":493,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":494,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'13","last_scrub_stamp":"2024-03-04T16:26:19.742509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:19.742509+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:33.843176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598278+0000","last_change":"2024-03-04T16:27:03.027523+0000","last_active":"2024-03-04T16:27:26.598278+0000","last_peered":"2024-03-04T16:27:26.598278+0000","last_clean":"2024-03-04T16:27:26.598278+0000","last_became_active":"2024-03-04T16:21:45.413068+0000","last_became_peered":"2024-03-04T16:21:45.413068+0000","last_unstale":"2024-03-04T16:27:26.598278+0000","last_undegraded":"2024-03-04T16:27:26.598278+0000","last_fullsized":"2024-03-04T16:27:26.598278+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:03.027421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.407279+0000","last_clean_scrub_stamp":"2024-03-04T16:27:03.027421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:11.685129+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"505'2","reported_seq":876,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596809+0000","last_change":"2024-03-04T16:26:16.707720+0000","last_active":"2024-03-04T16:27:26.596809+0000","last_peered":"2024-03-04T16:27:26.596809+0000","last_clean":"2024-03-04T16:27:26.596809+0000","last_became_active":"2024-03-04T16:25:41.956066+0000","last_became_peered":"2024-03-04T16:25:41.956066+0000","last_unstale":"2024-03-04T16:27:26.596809+0000","last_undegraded":"2024-03-04T16:27:26.596809+0000","last_fullsized":"2024-03-04T16:27:26.596809+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'2","last_scrub_stamp":"2024-03-04T16:26:16.707585+0000","last_deep_scrub":"505'2","last_deep_scrub_stamp":"2024-03-04T16:26:16.707585+0000","last_clean_scrub_stamp":"2024-03-04T16:26:16.707585+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:42.345591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'49","reported_seq":94,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511487+0000","last_change":"2024-03-04T16:27:00.740309+0000","last_active":"2024-03-04T16:27:26.511487+0000","last_peered":"2024-03-04T16:27:26.511487+0000","last_clean":"2024-03-04T16:27:26.511487+0000","last_became_active":"2024-03-04T16:25:41.963039+0000","last_became_peered":"2024-03-04T16:25:41.963039+0000","last_unstale":"2024-03-04T16:27:26.511487+0000","last_undegraded":"2024-03-04T16:27:26.511487+0000","last_fullsized":"2024-03-04T16:27:26.511487+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'49","last_scrub_stamp":"2024-03-04T16:27:00.740174+0000","last_deep_scrub":"90'21","last_deep_scrub_stamp":"2024-03-04T16:04:06.515823+0000","last_clean_scrub_stamp":"2024-03-04T16:27:00.740174+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:04.779080+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"517'13","reported_seq":128,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512342+0000","last_change":"2024-03-04T16:26:55.823122+0000","last_active":"2024-03-04T16:27:26.512342+0000","last_peered":"2024-03-04T16:27:26.512342+0000","last_clean":"2024-03-04T16:27:26.512342+0000","last_became_active":"2024-03-04T16:25:41.945789+0000","last_became_peered":"2024-03-04T16:25:41.945789+0000","last_unstale":"2024-03-04T16:27:26.512342+0000","last_undegraded":"2024-03-04T16:27:26.512342+0000","last_fullsized":"2024-03-04T16:27:26.512342+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"517'13","last_scrub_stamp":"2024-03-04T16:26:55.823038+0000","last_deep_scrub":"517'13","last_deep_scrub_stamp":"2024-03-04T16:26:55.823038+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.823038+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:23.287679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511449+0000","last_change":"2024-03-04T16:26:18.781428+0000","last_active":"2024-03-04T16:27:26.511449+0000","last_peered":"2024-03-04T16:27:26.511449+0000","last_clean":"2024-03-04T16:27:26.511449+0000","last_became_active":"2024-03-04T16:25:41.965545+0000","last_became_peered":"2024-03-04T16:25:41.965545+0000","last_unstale":"2024-03-04T16:27:26.511449+0000","last_undegraded":"2024-03-04T16:27:26.511449+0000","last_fullsized":"2024-03-04T16:27:26.511449+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:18.781302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:48.753581+0000","last_clean_scrub_stamp":"2024-03-04T16:26:18.781302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:38.039612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":434,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.513073+0000","last_change":"2024-03-04T16:26:48.837306+0000","last_active":"2024-03-04T16:27:26.513073+0000","last_peered":"2024-03-04T16:27:26.513073+0000","last_clean":"2024-03-04T16:27:26.513073+0000","last_became_active":"2024-03-04T16:26:16.732380+0000","last_became_peered":"2024-03-04T16:26:16.732380+0000","last_unstale":"2024-03-04T16:27:26.513073+0000","last_undegraded":"2024-03-04T16:27:26.513073+0000","last_fullsized":"2024-03-04T16:27:26.513073+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:48.837217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:06:49.140391+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.837217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:54.108872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'42","reported_seq":108,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512099+0000","last_change":"2024-03-04T16:27:06.921399+0000","last_active":"2024-03-04T16:27:26.512099+0000","last_peered":"2024-03-04T16:27:26.512099+0000","last_clean":"2024-03-04T16:27:26.512099+0000","last_became_active":"2024-03-04T16:25:35.913673+0000","last_became_peered":"2024-03-04T16:25:35.913673+0000","last_unstale":"2024-03-04T16:27:26.512099+0000","last_undegraded":"2024-03-04T16:27:26.512099+0000","last_fullsized":"2024-03-04T16:27:26.512099+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-04T16:27:06.921303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:27:06.921303+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:19.081502+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511304+0000","last_change":"2024-03-04T16:26:57.714648+0000","last_active":"2024-03-04T16:27:26.511304+0000","last_peered":"2024-03-04T16:27:26.511304+0000","last_clean":"2024-03-04T16:27:26.511304+0000","last_became_active":"2024-03-04T16:25:41.963328+0000","last_became_peered":"2024-03-04T16:25:41.963328+0000","last_unstale":"2024-03-04T16:27:26.511304+0000","last_undegraded":"2024-03-04T16:27:26.511304+0000","last_fullsized":"2024-03-04T16:27:26.511304+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:57.714536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:57.714536+0000","last_clean_scrub_stamp":"2024-03-04T16:26:57.714536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:59.664488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1010,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598082+0000","last_change":"2024-03-04T16:26:38.121206+0000","last_active":"2024-03-04T16:27:26.598082+0000","last_peered":"2024-03-04T16:27:26.598082+0000","last_clean":"2024-03-04T16:27:26.598082+0000","last_became_active":"2024-03-04T16:25:59.527386+0000","last_became_peered":"2024-03-04T16:25:59.527386+0000","last_unstale":"2024-03-04T16:27:26.598082+0000","last_undegraded":"2024-03-04T16:27:26.598082+0000","last_fullsized":"2024-03-04T16:27:26.598082+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:38.121069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:38.121069+0000","last_clean_scrub_stamp":"2024-03-04T16:26:38.121069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:55.746919+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"491'10","reported_seq":103,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512305+0000","last_change":"2024-03-04T16:26:55.718528+0000","last_active":"2024-03-04T16:27:26.512305+0000","last_peered":"2024-03-04T16:27:26.512305+0000","last_clean":"2024-03-04T16:27:26.512305+0000","last_became_active":"2024-03-04T16:25:41.961001+0000","last_became_peered":"2024-03-04T16:25:41.961001+0000","last_unstale":"2024-03-04T16:27:26.512305+0000","last_undegraded":"2024-03-04T16:27:26.512305+0000","last_fullsized":"2024-03-04T16:27:26.512305+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'10","last_scrub_stamp":"2024-03-04T16:26:55.718433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:46.732387+0000","last_clean_scrub_stamp":"2024-03-04T16:26:55.718433+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:05.895715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":82,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512227+0000","last_change":"2024-03-04T16:26:10.736176+0000","last_active":"2024-03-04T16:27:26.512227+0000","last_peered":"2024-03-04T16:27:26.512227+0000","last_clean":"2024-03-04T16:27:26.512227+0000","last_became_active":"2024-03-04T16:25:35.905845+0000","last_became_peered":"2024-03-04T16:25:35.905845+0000","last_unstale":"2024-03-04T16:27:26.512227+0000","last_undegraded":"2024-03-04T16:27:26.512227+0000","last_fullsized":"2024-03-04T16:27:26.512227+0000","mapping_epoch":488,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:10.736056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:42.653707+0000","last_clean_scrub_stamp":"2024-03-04T16:26:10.736056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'46","reported_seq":1157,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597084+0000","last_change":"2024-03-04T16:26:52.768796+0000","last_active":"2024-03-04T16:27:26.597084+0000","last_peered":"2024-03-04T16:27:26.597084+0000","last_clean":"2024-03-04T16:27:26.597084+0000","last_became_active":"2024-03-04T16:25:41.958609+0000","last_became_peered":"2024-03-04T16:25:41.958609+0000","last_unstale":"2024-03-04T16:27:26.597084+0000","last_undegraded":"2024-03-04T16:27:26.597084+0000","last_fullsized":"2024-03-04T16:27:26.597084+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-04T16:26:52.768648+0000","last_deep_scrub":"441'46","last_deep_scrub_stamp":"2024-03-04T16:25:18.623813+0000","last_clean_scrub_stamp":"2024-03-04T16:26:52.768648+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:06.856507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"507'2","reported_seq":250,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512736+0000","last_change":"2024-03-04T16:26:20.703392+0000","last_active":"2024-03-04T16:27:26.512736+0000","last_peered":"2024-03-04T16:27:26.512736+0000","last_clean":"2024-03-04T16:27:26.512736+0000","last_became_active":"2024-03-04T16:26:16.733440+0000","last_became_peered":"2024-03-04T16:26:16.733440+0000","last_unstale":"2024-03-04T16:27:26.512736+0000","last_undegraded":"2024-03-04T16:27:26.512736+0000","last_fullsized":"2024-03-04T16:27:26.512736+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'2","last_scrub_stamp":"2024-03-04T16:26:20.703307+0000","last_deep_scrub":"507'2","last_deep_scrub_stamp":"2024-03-04T16:26:20.703307+0000","last_clean_scrub_stamp":"2024-03-04T16:26:20.703307+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:48.454458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1028,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.598752+0000","last_change":"2024-03-04T16:27:07.064135+0000","last_active":"2024-03-04T16:27:26.598752+0000","last_peered":"2024-03-04T16:27:26.598752+0000","last_clean":"2024-03-04T16:27:26.598752+0000","last_became_active":"2024-03-04T16:25:41.969123+0000","last_became_peered":"2024-03-04T16:25:41.969123+0000","last_unstale":"2024-03-04T16:27:26.598752+0000","last_undegraded":"2024-03-04T16:27:26.598752+0000","last_fullsized":"2024-03-04T16:27:26.598752+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.064036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:27:07.064036+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.064036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:20.800314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":78,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511424+0000","last_change":"2024-03-04T16:26:56.709477+0000","last_active":"2024-03-04T16:27:26.511424+0000","last_peered":"2024-03-04T16:27:26.511424+0000","last_clean":"2024-03-04T16:27:26.511424+0000","last_became_active":"2024-03-04T16:25:41.966246+0000","last_became_peered":"2024-03-04T16:25:41.966246+0000","last_unstale":"2024-03-04T16:27:26.511424+0000","last_undegraded":"2024-03-04T16:27:26.511424+0000","last_fullsized":"2024-03-04T16:27:26.511424+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:56.709331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:56.709331+0000","last_clean_scrub_stamp":"2024-03-04T16:26:56.709331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:12.623820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'32","reported_seq":108,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512177+0000","last_change":"2024-03-04T16:26:29.705756+0000","last_active":"2024-03-04T16:27:26.512177+0000","last_peered":"2024-03-04T16:27:26.512177+0000","last_clean":"2024-03-04T16:27:26.512177+0000","last_became_active":"2024-03-04T16:25:59.525685+0000","last_became_peered":"2024-03-04T16:25:59.525685+0000","last_unstale":"2024-03-04T16:27:26.512177+0000","last_undegraded":"2024-03-04T16:27:26.512177+0000","last_fullsized":"2024-03-04T16:27:26.512177+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-04T16:26:29.705574+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-04T16:26:29.705574+0000","last_clean_scrub_stamp":"2024-03-04T16:26:29.705574+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:58.114809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":456,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510547+0000","last_change":"2024-03-04T16:26:59.032019+0000","last_active":"2024-03-04T16:27:26.510547+0000","last_peered":"2024-03-04T16:27:26.510547+0000","last_clean":"2024-03-04T16:27:26.510547+0000","last_became_active":"2024-03-04T16:25:41.958623+0000","last_became_peered":"2024-03-04T16:25:41.958623+0000","last_unstale":"2024-03-04T16:27:26.510547+0000","last_undegraded":"2024-03-04T16:27:26.510547+0000","last_fullsized":"2024-03-04T16:27:26.510547+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:59.031905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:32.684717+0000","last_clean_scrub_stamp":"2024-03-04T16:26:59.031905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:19.598491+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1023,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.596738+0000","last_change":"2024-03-04T16:26:48.646715+0000","last_active":"2024-03-04T16:27:26.596738+0000","last_peered":"2024-03-04T16:27:26.596738+0000","last_clean":"2024-03-04T16:27:26.596738+0000","last_became_active":"2024-03-04T16:25:41.957803+0000","last_became_peered":"2024-03-04T16:25:41.957803+0000","last_unstale":"2024-03-04T16:27:26.596738+0000","last_undegraded":"2024-03-04T16:27:26.596738+0000","last_fullsized":"2024-03-04T16:27:26.596738+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:48.646572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:48.646572+0000","last_clean_scrub_stamp":"2024-03-04T16:26:48.646572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:17.653663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":299,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.597840+0000","last_change":"2024-03-04T16:27:22.631026+0000","last_active":"2024-03-04T16:27:26.597840+0000","last_peered":"2024-03-04T16:27:26.597840+0000","last_clean":"2024-03-04T16:27:26.597840+0000","last_became_active":"2024-03-04T16:25:59.526640+0000","last_became_peered":"2024-03-04T16:25:59.526640+0000","last_unstale":"2024-03-04T16:27:26.597840+0000","last_undegraded":"2024-03-04T16:27:26.597840+0000","last_fullsized":"2024-03-04T16:27:26.597840+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:22.630920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:04:07.762953+0000","last_clean_scrub_stamp":"2024-03-04T16:27:22.630920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:27.299972+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":96,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512234+0000","last_change":"2024-03-04T16:26:32.725623+0000","last_active":"2024-03-04T16:27:26.512234+0000","last_peered":"2024-03-04T16:27:26.512234+0000","last_clean":"2024-03-04T16:27:26.512234+0000","last_became_active":"2024-03-04T16:25:41.960919+0000","last_became_peered":"2024-03-04T16:25:41.960919+0000","last_unstale":"2024-03-04T16:27:26.512234+0000","last_undegraded":"2024-03-04T16:27:26.512234+0000","last_fullsized":"2024-03-04T16:27:26.512234+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-04T16:26:32.725509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:02:44.697356+0000","last_clean_scrub_stamp":"2024-03-04T16:26:32.725509+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:57.100907+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512274+0000","last_change":"2024-03-04T16:27:07.865879+0000","last_active":"2024-03-04T16:27:26.512274+0000","last_peered":"2024-03-04T16:27:26.512274+0000","last_clean":"2024-03-04T16:27:26.512274+0000","last_became_active":"2024-03-04T16:25:59.522487+0000","last_became_peered":"2024-03-04T16:25:59.522487+0000","last_unstale":"2024-03-04T16:27:26.512274+0000","last_undegraded":"2024-03-04T16:27:26.512274+0000","last_fullsized":"2024-03-04T16:27:26.512274+0000","mapping_epoch":495,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":496,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:27:07.865756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:25:56.691335+0000","last_clean_scrub_stamp":"2024-03-04T16:27:07.865756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:10.941362+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"338'11","reported_seq":517,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.510636+0000","last_change":"2024-03-04T16:26:36.698181+0000","last_active":"2024-03-04T16:27:26.510636+0000","last_peered":"2024-03-04T16:27:26.510636+0000","last_clean":"2024-03-04T16:27:26.510636+0000","last_became_active":"2024-03-04T16:19:04.756013+0000","last_became_peered":"2024-03-04T16:19:04.756013+0000","last_unstale":"2024-03-04T16:27:26.510636+0000","last_undegraded":"2024-03-04T16:27:26.510636+0000","last_fullsized":"2024-03-04T16:27:26.510636+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'11","last_scrub_stamp":"2024-03-04T16:26:36.698080+0000","last_deep_scrub":"90'9","last_deep_scrub_stamp":"2024-03-04T16:05:35.758554+0000","last_clean_scrub_stamp":"2024-03-04T16:26:36.698080+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:52.703568+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":130,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.512604+0000","last_change":"2024-03-04T16:26:41.754214+0000","last_active":"2024-03-04T16:27:26.512604+0000","last_peered":"2024-03-04T16:27:26.512604+0000","last_clean":"2024-03-04T16:27:26.512604+0000","last_became_active":"2024-03-04T16:26:16.733370+0000","last_became_peered":"2024-03-04T16:26:16.733370+0000","last_unstale":"2024-03-04T16:27:26.512604+0000","last_undegraded":"2024-03-04T16:27:26.512604+0000","last_fullsized":"2024-03-04T16:27:26.512604+0000","mapping_epoch":505,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":506,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-04T16:26:41.754075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-04T16:26:41.754075+0000","last_clean_scrub_stamp":"2024-03-04T16:26:41.754075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:28:02.504263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":90,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-04T16:27:26.511736+0000","last_change":"2024-03-04T16:26:25.764118+0000","last_active":"2024-03-04T16:27:26.511736+0000","last_peered":"2024-03-04T16:27:26.511736+0000","last_clean":"2024-03-04T16:27:26.511736+0000","last_became_active":"2024-03-04T16:25:41.957322+0000","last_became_peered":"2024-03-04T16:25:41.957322+0000","last_unstale":"2024-03-04T16:27:26.511736+0000","last_undegraded":"2024-03-04T16:27:26.511736+0000","last_fullsized":"2024-03-04T16:27:26.511736+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-04T16:26:25.764002+0000","last_deep_scrub":"318'25","last_deep_scrub_stamp":"2024-03-04T16:18:14.650758+0000","last_clean_scrub_stamp":"2024-03-04T16:26:25.764002+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-04T16:27:50.875243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1471,"num_read_kb":1466,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4132576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":4137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1519,"num_read_kb":1276,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":14974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1058,"ondisk_log_size":1058,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":779,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070766,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957196,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92787956,"statfs":{"total":95995035648,"available":95014866944,"internally_reserved":0,"allocated":1441792,"data_stored":1014437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":978713034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365106,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":433784,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93311368,"statfs":{"total":95995035648,"available":95550840832,"internally_reserved":0,"allocated":1417216,"data_stored":1012125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128804,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":1127934,"kb_avail":92616348,"statfs":{"total":95995035648,"available":94839140352,"internally_reserved":0,"allocated":872448,"data_stored":423199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1155004869},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738704,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":925260,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92819892,"statfs":{"total":95995035648,"available":95047569408,"internally_reserved":0,"allocated":1441792,"data_stored":1013781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411654,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":671996,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":93073156,"statfs":{"total":95995035648,"available":95306911744,"internally_reserved":0,"allocated":831488,"data_stored":422587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150017,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1202716,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":1201854,"kb_avail":92542436,"statfs":{"total":95995035648,"available":94763454464,"internally_reserved":0,"allocated":864256,"data_stored":423523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":1230698952},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953788,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":740040,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":93005112,"statfs":{"total":95995035648,"available":95237234688,"internally_reserved":0,"allocated":847872,"data_stored":422637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":756939205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626744,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032952,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":1032126,"kb_avail":92712200,"statfs":{"total":95995035648,"available":94937292800,"internally_reserved":0,"allocated":831488,"data_stored":422193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1056897478},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-04T16:27:33.681 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-04T16:27:33.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282 shell --fsid c453baf0-da3f-11ee-95c6-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-04T16:27:33.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:27:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug there is no tcmu-runner data available 2024-03-04T16:27:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:33 smithi176 ceph-mon[33619]: pgmap v1169: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-04T16:27:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:33 smithi176 ceph-mon[33619]: from='client.36347 v1:172.21.15.38:0/746183218' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-04T16:27:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:33 smithi176 ceph-mon[33619]: 5.15 scrub starts 2024-03-04T16:27:33.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:33 smithi176 ceph-mon[33619]: 5.15 scrub ok 2024-03-04T16:27:33.930 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/mon.a/config 2024-03-04T16:27:34.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: 3.8 scrub starts 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: 3.8 scrub ok 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: 6.8 scrub starts 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[29779]: 6.8 scrub ok 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: 3.8 scrub starts 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: 3.8 scrub ok 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: 6.8 scrub starts 2024-03-04T16:27:34.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:34 smithi038 ceph-mon[38113]: 6.8 scrub ok 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: 3.8 scrub starts 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: 3.8 scrub ok 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: from='client.24463 v1:172.21.15.176:0/2233515072' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: pgmap v1170: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: 6.8 scrub starts 2024-03-04T16:27:34.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:34 smithi176 ceph-mon[33619]: 6.8 scrub ok 2024-03-04T16:27:35.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[38113]: 5.b deep-scrub starts 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[38113]: 5.b deep-scrub ok 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[38113]: 6.1f scrub starts 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[38113]: 6.1f scrub ok 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[29779]: 5.b deep-scrub starts 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[29779]: 5.b deep-scrub ok 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[29779]: 6.1f scrub starts 2024-03-04T16:27:35.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-mon[29779]: 6.1f scrub ok 2024-03-04T16:27:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:35 smithi176 ceph-mon[33619]: 5.b deep-scrub starts 2024-03-04T16:27:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:35 smithi176 ceph-mon[33619]: 5.b deep-scrub ok 2024-03-04T16:27:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:35 smithi176 ceph-mon[33619]: 6.1f scrub starts 2024-03-04T16:27:35.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:35 smithi176 ceph-mon[33619]: 6.1f scrub ok 2024-03-04T16:27:36.242 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:35] ENGINE Bus STOPPING 2024-03-04T16:27:36.242 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:27:36.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:35] ENGINE Bus STOPPED 2024-03-04T16:27:36.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:35 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:35] ENGINE Bus STARTING 2024-03-04T16:27:36.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:36] ENGINE Serving on http://:::9283 2024-03-04T16:27:36.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:36] ENGINE Bus STARTED 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 5.12 scrub starts 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 5.12 scrub ok 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 5.1e scrub starts 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 5.1e scrub ok 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 3.d scrub starts 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: 3.d scrub ok 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='client.? v1:172.21.15.38:0/749409411' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='client.36353 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='client.36353 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:27:36.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[29779]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 5.12 scrub starts 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 5.12 scrub ok 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 5.1e scrub starts 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 5.1e scrub ok 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 3.d scrub starts 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: 3.d scrub ok 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='client.? v1:172.21.15.38:0/749409411' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.744 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='client.36353 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='client.36353 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-04T16:27:36.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:27:36.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:27:36.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:27:36.745 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:36 smithi038 ceph-mon[38113]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 5.12 scrub starts 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 5.12 scrub ok 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 5.1e scrub starts 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 5.1e scrub ok 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 3.d scrub starts 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: 3.d scrub ok 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='client.? v1:172.21.15.38:0/749409411' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='client.36353 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='client.36353 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-04T16:27:36.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='mgr.34496 v1:172.21.15.38:0/4251858210' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-04T16:27:36.859 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:36 smithi176 ceph-mon[33619]: from='mgr.34496 ' entity='mgr.y' 2024-03-04T16:27:37.081 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-04T16:27:37.091 INFO:tasks.cephadm:Teardown begin 2024-03-04T16:27:37.092 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:27:37.150 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:27:37.208 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-04T16:27:37.208 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:27:37.248 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-04T16:27:37.268 INFO:tasks.cephadm:Stopping all daemons... 2024-03-04T16:27:37.268 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-04T16:27:37.269 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a 2024-03-04T16:27:37.631 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 systemd[1]: Stopping Ceph mon.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:37.631 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-mon[29779]: 3.16 scrub starts 2024-03-04T16:27:37.631 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-mon[29779]: 3.16 scrub ok 2024-03-04T16:27:37.631 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-mon[38113]: 3.16 scrub starts 2024-03-04T16:27:37.631 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-mon[38113]: 3.16 scrub ok 2024-03-04T16:27:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:37 smithi176 ceph-mon[33619]: 3.16 scrub starts 2024-03-04T16:27:37.858 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:37 smithi176 ceph-mon[33619]: 3.16 scrub ok 2024-03-04T16:27:37.992 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:27:37.630+0000 7ff2a8ed7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a[29775]: 2024-03-04T16:27:37.630+0000 7ff2a8ed7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-04T16:27:37.993 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:37 smithi038 podman[212517]: 2024-03-04 16:27:37.82507872 +0000 UTC m=+0.324060768 container died 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T16:27:38.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:38 smithi038 podman[212517]: 2024-03-04 16:27:38.290054548 +0000 UTC m=+0.789036599 container cleanup 25f9cd8cc85672e2f51d2c14f4f13fb91fb3e795a3a0029cbf2c61043b3b8ae0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-04T16:27:38.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 04 16:27:38 smithi038 bash[212517]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-a 2024-03-04T16:27:38.721 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.a.service' 2024-03-04T16:27:38.776 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:38.776 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-04T16:27:38.777 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-04T16:27:38.777 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.c 2024-03-04T16:27:39.061 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:38 smithi038 systemd[1]: Stopping Ceph mon.c for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:39.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c[38107]: 2024-03-04T16:27:39.061+0000 7f1c26850700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:39.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c[38107]: 2024-03-04T16:27:39.061+0000 7f1c26850700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-04T16:27:39.487 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:39 smithi038 podman[212637]: 2024-03-04 16:27:39.153502714 +0000 UTC m=+0.195139666 container died 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE Bus STOPPING 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE Bus STOPPED 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE Bus STARTING 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE Serving on http://:::9283 2024-03-04T16:27:39.488 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:39 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: [04/Mar/2024:16:27:39] ENGINE Bus STARTED 2024-03-04T16:27:39.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:39 smithi038 podman[212637]: 2024-03-04 16:27:39.603844238 +0000 UTC m=+0.645481188 container cleanup 7f12cd7aedbf6842ce4243ee6bacb2ea59b1caa3514d28fe5a9efa4d44ec4a46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-04T16:27:39.742 INFO:journalctl@ceph.mon.c.smithi038.stdout:Mar 04 16:27:39 smithi038 bash[212637]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-c 2024-03-04T16:27:40.074 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.c.service' 2024-03-04T16:27:40.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:40.111 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-04T16:27:40.112 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-04T16:27:40.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.b 2024-03-04T16:27:40.685 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:40 smithi176 systemd[1]: Stopping Ceph mon.b for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:41.069 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b[33604]: 2024-03-04T16:27:40.683+0000 7fce81104700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:41.069 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b[33604]: 2024-03-04T16:27:40.683+0000 7fce81104700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-04T16:27:41.069 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:40 smithi176 podman[156429]: 2024-03-04 16:27:40.786955669 +0000 UTC m=+0.202253873 container died f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T16:27:41.357 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:41 smithi176 podman[156429]: 2024-03-04 16:27:41.167721921 +0000 UTC m=+0.583020133 container cleanup f2376ff95ea7eb2d0d9659118aff81198e2df4c129d457033cf16ea603401ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-04T16:27:41.358 INFO:journalctl@ceph.mon.b.smithi176.stdout:Mar 04 16:27:41 smithi176 bash[156429]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mon-b 2024-03-04T16:27:41.581 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mon.b.service' 2024-03-04T16:27:41.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:41.635 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-04T16:27:41.636 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-04T16:27:41.636 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y 2024-03-04T16:27:41.958 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:41 smithi038 systemd[1]: Stopping Ceph mgr.y for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:42.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:27:41.958+0000 7fcebfed7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:42.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:41 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y[30010]: 2024-03-04T16:27:41.958+0000 7fcebfed7700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-04T16:27:42.243 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:41 smithi038 podman[212768]: 2024-03-04 16:27:41.991139899 +0000 UTC m=+0.158533370 container died 5d1ee81947b483bf2e552e81877c03effbbfc053a498cd59d51d08d9527a4460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T16:27:42.635 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:42 smithi038 podman[212768]: 2024-03-04 16:27:42.367565176 +0000 UTC m=+0.534958640 container cleanup 5d1ee81947b483bf2e552e81877c03effbbfc053a498cd59d51d08d9527a4460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9ca0b5de35ed230e84387abdd8e619ed8e4f3282, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y, ceph=True, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:27:42.636 INFO:journalctl@ceph.mgr.y.smithi038.stdout:Mar 04 16:27:42 smithi038 bash[212768]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-y 2024-03-04T16:27:42.792 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-04T16:27:42.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:42.829 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-04T16:27:42.830 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-04T16:27:42.830 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.x 2024-03-04T16:27:43.169 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:27:42 smithi176 systemd[1]: Stopping Ceph mgr.x for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:43.513 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:27:43 smithi176 podman[156554]: 2024-03-04 16:27:43.169564756 +0000 UTC m=+0.145849560 container died fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T16:27:43.831 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:27:43 smithi176 podman[156554]: 2024-03-04 16:27:43.513701683 +0000 UTC m=+0.489986485 container cleanup fd047dad626a963728e5ed10d33539eb88d4d31e1b193e1ad697a23e3102bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-04T16:27:43.831 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Mar 04 16:27:43 smithi176 bash[156554]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-mgr-x 2024-03-04T16:27:43.942 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-04T16:27:43.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:43.979 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-04T16:27:43.979 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-04T16:27:43.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.0 2024-03-04T16:27:44.288 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:44 smithi038 systemd[1]: Stopping Ceph osd.0 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:44.742 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:44 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0[42115]: 2024-03-04T16:27:44.287+0000 7f6099bdd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:44.743 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:44 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0[42115]: 2024-03-04T16:27:44.287+0000 7f6099bdd700 -1 osd.0 521 *** Got signal Terminated *** 2024-03-04T16:27:44.743 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:44 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0[42115]: 2024-03-04T16:27:44.287+0000 7f6099bdd700 -1 osd.0 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:27:50.147 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:49 smithi038 podman[212888]: 2024-03-04 16:27:49.883557679 +0000 UTC m=+5.708733579 container died 1e7beff76f8b8aaccfb0423527af6493a454fa3e5549f775d3405ea9c9835f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-04T16:27:50.691 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:50 smithi038 podman[212888]: 2024-03-04 16:27:50.414357275 +0000 UTC m=+6.239533174 container cleanup 1e7beff76f8b8aaccfb0423527af6493a454fa3e5549f775d3405ea9c9835f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-04T16:27:50.691 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:50 smithi038 bash[212888]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0 2024-03-04T16:27:50.992 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:50 smithi038 podman[212905]: 2024-03-04 16:27:50.691186332 +0000 UTC m=+0.803147604 container remove 1e7beff76f8b8aaccfb0423527af6493a454fa3e5549f775d3405ea9c9835f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-04T16:27:51.344 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:50 smithi038 podman[212998]: 2024-03-04 16:27:50.90043845 +0000 UTC m=+0.017882368 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:27:51.344 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:51 smithi038 podman[212998]: 2024-03-04 16:27:51.008792776 +0000 UTC m=+0.126236716 container create f6014d43733978bfaa2c427087c28543099bb28bd04504fbaaac3c14a8f79408 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T16:27:51.344 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:51 smithi038 podman[212998]: 2024-03-04 16:27:51.227916929 +0000 UTC m=+0.345360840 container init f6014d43733978bfaa2c427087c28543099bb28bd04504fbaaac3c14a8f79408 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:27:51.344 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:51 smithi038 podman[212998]: 2024-03-04 16:27:51.230995984 +0000 UTC m=+0.348439883 container start f6014d43733978bfaa2c427087c28543099bb28bd04504fbaaac3c14a8f79408 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:27:51.344 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:51 smithi038 podman[212998]: 2024-03-04 16:27:51.344810632 +0000 UTC m=+0.462254576 container attach f6014d43733978bfaa2c427087c28543099bb28bd04504fbaaac3c14a8f79408 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-04T16:27:52.742 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 04 16:27:52 smithi038 podman[212998]: 2024-03-04 16:27:52.542074563 +0000 UTC m=+1.659518463 container died f6014d43733978bfaa2c427087c28543099bb28bd04504fbaaac3c14a8f79408 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS) 2024-03-04T16:27:53.425 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.0.service' 2024-03-04T16:27:53.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:27:53.467 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-04T16:27:53.468 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-04T16:27:53.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.1 2024-03-04T16:27:53.992 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:53 smithi038 systemd[1]: Stopping Ceph osd.1 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:27:53.992 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1[47947]: 2024-03-04T16:27:53.796+0000 7f04302aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:27:53.992 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1[47947]: 2024-03-04T16:27:53.796+0000 7f04302aa700 -1 osd.1 521 *** Got signal Terminated *** 2024-03-04T16:27:53.993 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:53 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1[47947]: 2024-03-04T16:27:53.796+0000 7f04302aa700 -1 osd.1 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:27:59.742 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:59 smithi038 podman[213116]: 2024-03-04 16:27:59.467199438 +0000 UTC m=+5.770199068 container died bc013cfaafe376582db150a39b1ec19216e6f082697ec8abaf30782a36756ea6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:28:00.232 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:59 smithi038 podman[213116]: 2024-03-04 16:27:59.955649515 +0000 UTC m=+6.258649145 container cleanup bc013cfaafe376582db150a39b1ec19216e6f082697ec8abaf30782a36756ea6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-04T16:28:00.232 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:27:59 smithi038 bash[213116]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1 2024-03-04T16:28:00.492 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213133]: 2024-03-04 16:28:00.232557539 +0000 UTC m=+0.758892181 container remove bc013cfaafe376582db150a39b1ec19216e6f082697ec8abaf30782a36756ea6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:28:00.853 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213226]: 2024-03-04 16:28:00.437555986 +0000 UTC m=+0.017643171 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:00.853 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213226]: 2024-03-04 16:28:00.550238054 +0000 UTC m=+0.130325216 container create 394b710f004aef6e2ab445445c27091509ec96930f3b4a3301b131492131ba79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:28:00.853 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213226]: 2024-03-04 16:28:00.761125218 +0000 UTC m=+0.341212394 container init 394b710f004aef6e2ab445445c27091509ec96930f3b4a3301b131492131ba79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-04T16:28:00.853 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213226]: 2024-03-04 16:28:00.764765047 +0000 UTC m=+0.344852214 container start 394b710f004aef6e2ab445445c27091509ec96930f3b4a3301b131492131ba79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:28:00.853 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:00 smithi038 podman[213226]: 2024-03-04 16:28:00.852773179 +0000 UTC m=+0.432860368 container attach 394b710f004aef6e2ab445445c27091509ec96930f3b4a3301b131492131ba79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T16:28:01.242 INFO:journalctl@ceph.osd.1.smithi038.stdout:Mar 04 16:28:01 smithi038 podman[213226]: 2024-03-04 16:28:01.069071539 +0000 UTC m=+0.649158707 container died 394b710f004aef6e2ab445445c27091509ec96930f3b4a3301b131492131ba79 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-04T16:28:01.778 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.1.service' 2024-03-04T16:28:01.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:01.825 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-04T16:28:01.826 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-04T16:28:01.826 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.2 2024-03-04T16:28:02.152 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:01 smithi038 systemd[1]: Stopping Ceph osd.2 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:02.495 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:02 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2[53987]: 2024-03-04T16:28:02.151+0000 7f3f26c46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:02.496 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:02 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2[53987]: 2024-03-04T16:28:02.151+0000 7f3f26c46700 -1 osd.2 521 *** Got signal Terminated *** 2024-03-04T16:28:02.496 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:02 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2[53987]: 2024-03-04T16:28:02.151+0000 7f3f26c46700 -1 osd.2 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:08.104 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:07 smithi038 podman[213332]: 2024-03-04 16:28:07.780934889 +0000 UTC m=+5.760367460 container died a7c98636834c33d7a44a315ae5e0a77d030e8931c4d34809953a5db703136007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:28:08.492 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:08 smithi038 podman[213332]: 2024-03-04 16:28:08.285988403 +0000 UTC m=+6.265421015 container cleanup a7c98636834c33d7a44a315ae5e0a77d030e8931c4d34809953a5db703136007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T16:28:08.493 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:08 smithi038 bash[213332]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2 2024-03-04T16:28:08.826 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:08 smithi038 podman[213349]: 2024-03-04 16:28:08.52063914 +0000 UTC m=+0.737509424 container remove a7c98636834c33d7a44a315ae5e0a77d030e8931c4d34809953a5db703136007 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:28:09.132 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:08 smithi038 podman[213443]: 2024-03-04 16:28:08.729278544 +0000 UTC m=+0.017682966 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:09.132 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:08 smithi038 podman[213443]: 2024-03-04 16:28:08.829796116 +0000 UTC m=+0.118200538 container create b5eb92ced4350ca1fd860c63a3687b8b255f0758349dfafdabe1db510cbb3260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-04T16:28:09.132 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:09 smithi038 podman[213443]: 2024-03-04 16:28:09.04114502 +0000 UTC m=+0.329549444 container init b5eb92ced4350ca1fd860c63a3687b8b255f0758349dfafdabe1db510cbb3260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:28:09.132 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:09 smithi038 podman[213443]: 2024-03-04 16:28:09.044251654 +0000 UTC m=+0.332656069 container start b5eb92ced4350ca1fd860c63a3687b8b255f0758349dfafdabe1db510cbb3260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD) 2024-03-04T16:28:09.132 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:09 smithi038 podman[213443]: 2024-03-04 16:28:09.132642852 +0000 UTC m=+0.421047267 container attach b5eb92ced4350ca1fd860c63a3687b8b255f0758349dfafdabe1db510cbb3260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T16:28:09.492 INFO:journalctl@ceph.osd.2.smithi038.stdout:Mar 04 16:28:09 smithi038 podman[213443]: 2024-03-04 16:28:09.34122246 +0000 UTC m=+0.629626876 container died b5eb92ced4350ca1fd860c63a3687b8b255f0758349dfafdabe1db510cbb3260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, GIT_CLEAN=True, ceph=True) 2024-03-04T16:28:09.898 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.2.service' 2024-03-04T16:28:09.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:09.943 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-04T16:28:09.943 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-04T16:28:09.943 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.3 2024-03-04T16:28:10.230 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:09 smithi038 systemd[1]: Stopping Ceph osd.3 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:10.492 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:10.230+0000 7ff1ca12c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:10.492 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:10.230+0000 7ff1ca12c700 -1 osd.3 521 *** Got signal Terminated *** 2024-03-04T16:28:10.492 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:10 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:10.230+0000 7ff1ca12c700 -1 osd.3 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:11.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:10 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:10.686+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:12.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:11 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:11.840+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:12.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:11 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:11.649+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:12.742 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:12 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:12.300+0000 7ff1c2718700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.997672+0000 front 2024-03-04T16:27:46.997712+0000 (oldest deadline 2024-03-04T16:28:12.297848+0000) 2024-03-04T16:28:13.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:12.617+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:13.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:12.660+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:13.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:12 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:12.829+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:13.742 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:13 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:13.315+0000 7ff1c2718700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.997672+0000 front 2024-03-04T16:27:46.997712+0000 (oldest deadline 2024-03-04T16:28:12.297848+0000) 2024-03-04T16:28:13.857 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:13.844+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:13.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:13.603+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:13.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:13.739+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:13.858 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:13 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:13.651+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:14.742 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:14 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3[60060]: 2024-03-04T16:28:14.347+0000 7ff1c2718700 -1 osd.3 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.997672+0000 front 2024-03-04T16:27:46.997712+0000 (oldest deadline 2024-03-04T16:28:12.297848+0000) 2024-03-04T16:28:15.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:14.652+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:15.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:14.803+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:15.109 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:14.638+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:15.109 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:14 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:14.785+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:16.107 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:15.616+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:16.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:15.741+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:16.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:15.638+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:16.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:15 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:15.842+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:16.114 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:15 smithi038 podman[213548]: 2024-03-04 16:28:15.836162245 +0000 UTC m=+5.708303984 container died 6ed39afa375551b4420afe9b3c5013a8127fd617e0b184692aeedbd9f771440e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:28:16.492 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:16 smithi038 podman[213548]: 2024-03-04 16:28:16.281230569 +0000 UTC m=+6.153372350 container cleanup 6ed39afa375551b4420afe9b3c5013a8127fd617e0b184692aeedbd9f771440e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-04T16:28:16.492 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:16 smithi038 bash[213548]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3 2024-03-04T16:28:16.956 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:16 smithi038 podman[213566]: 2024-03-04 16:28:16.599377233 +0000 UTC m=+0.765273424 container remove 6ed39afa375551b4420afe9b3c5013a8127fd617e0b184692aeedbd9f771440e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-04T16:28:17.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:16.813+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:17.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:16.634+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:17.109 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:16.693+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:17.109 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:16 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:16.661+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:17.242 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:16 smithi038 podman[213657]: 2024-03-04 16:28:16.860618467 +0000 UTC m=+0.018588008 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:17.243 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:16 smithi038 podman[213657]: 2024-03-04 16:28:16.958515681 +0000 UTC m=+0.116485247 container create 20028a5729e56cf66dd5bf84a7c54e9aad4724ad99a4632511ce6e8e26220a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:28:17.243 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:17 smithi038 podman[213657]: 2024-03-04 16:28:17.18593615 +0000 UTC m=+0.343905677 container init 20028a5729e56cf66dd5bf84a7c54e9aad4724ad99a4632511ce6e8e26220a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T16:28:17.243 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:17 smithi038 podman[213657]: 2024-03-04 16:28:17.189105283 +0000 UTC m=+0.347074813 container start 20028a5729e56cf66dd5bf84a7c54e9aad4724ad99a4632511ce6e8e26220a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-04T16:28:17.743 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:17 smithi038 podman[213657]: 2024-03-04 16:28:17.294570368 +0000 UTC m=+0.452539900 container attach 20028a5729e56cf66dd5bf84a7c54e9aad4724ad99a4632511ce6e8e26220a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222) 2024-03-04T16:28:17.743 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 04 16:28:17 smithi038 podman[213657]: 2024-03-04 16:28:17.493558951 +0000 UTC m=+0.651528486 container died 20028a5729e56cf66dd5bf84a7c54e9aad4724ad99a4632511ce6e8e26220a57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240222, GIT_BRANCH=HEAD, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-04T16:28:18.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:17.693+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:18.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:17.855+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:18.109 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:17.613+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:18.109 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:17 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:17.669+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:18.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.3.service' 2024-03-04T16:28:18.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:18.307 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-04T16:28:18.308 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-04T16:28:18.308 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.4 2024-03-04T16:28:18.606 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:18 smithi176 systemd[1]: Stopping Ceph osd.4 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:18.858 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:18.818+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:18.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:18.638+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:18.859 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:18.669+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:18.859 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:18.606+0000 7f591a864700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:18.859 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:18.606+0000 7f591a864700 -1 osd.4 521 *** Got signal Terminated *** 2024-03-04T16:28:18.859 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:18.606+0000 7f591a864700 -1 osd.4 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:18.859 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:18 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:18.676+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:19.858 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:19.704+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:19.858 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:19.849+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:19.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:19.631+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:19.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:19.602+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:19.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:19 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:19.602+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:20.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:20.607+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:20.858 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:20.722+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:20.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:20.577+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:20.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:20.577+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:21.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:20.895+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:21.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:20 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:20.895+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:21.813 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:21.583+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:21.813 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:21.764+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:21.813 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:21.764+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:54.941476+0000 front 2024-03-04T16:27:54.941441+0000 (oldest deadline 2024-03-04T16:28:20.841339+0000) 2024-03-04T16:28:21.813 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:21.555+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:21.813 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:21.555+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:22.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:21.902+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:22.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:21 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:21.902+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:22.741 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:22.741+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:22.741 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:22.571+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:22.741 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:22.571+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:22.741 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:22.594+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:22.741 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:22.594+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:23.107 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:22.741+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:54.941476+0000 front 2024-03-04T16:27:54.941441+0000 (oldest deadline 2024-03-04T16:28:20.841339+0000) 2024-03-04T16:28:23.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:22.882+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:23.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:22 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:22.882+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:23.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:23.535+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:23.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:23.535+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:23.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:23.613+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:23.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:23.613+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:24.195 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:23.909+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:24.196 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:23 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:23.909+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: *** Caught signal (Segmentation fault) ** 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: in thread 7f5905e36700 thread_name:safe_timer 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5921cbad20] 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55ca57688cca] 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55ca57716b58] 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55ca572fbda3] 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55ca57587d01] 2024-03-04T16:28:24.197 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55ca57276974] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55ca572789ca] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55ca5727a933] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55ca571dc56a] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55ca57836fef] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55ca57838311] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5921cb01ca] 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 13: clone() 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:24.076+0000 7f5905e36700 -1 *** Caught signal (Segmentation fault) ** 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: in thread 7f5905e36700 thread_name:safe_timer 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T16:28:24.198 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5921cbad20] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55ca57688cca] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55ca57716b58] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55ca572fbda3] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55ca57587d01] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55ca57276974] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55ca572789ca] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55ca5727a933] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55ca571dc56a] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55ca57836fef] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55ca57838311] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5921cb01ca] 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 13: clone() 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T16:28:24.199 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -3054> 2024-03-04T16:28:12.660+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -2942> 2024-03-04T16:28:13.651+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1905> 2024-03-04T16:28:14.652+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1739> 2024-03-04T16:28:15.638+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1579> 2024-03-04T16:28:16.661+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1512> 2024-03-04T16:28:17.693+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1451> 2024-03-04T16:28:18.606+0000 7f591a864700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1450> 2024-03-04T16:28:18.606+0000 7f591a864700 -1 osd.4 521 *** Got signal Terminated *** 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1448> 2024-03-04T16:28:18.606+0000 7f591a864700 -1 osd.4 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1445> 2024-03-04T16:28:18.676+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1364> 2024-03-04T16:28:19.704+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1259> 2024-03-04T16:28:20.722+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.200 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1194> 2024-03-04T16:28:21.764+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -1193> 2024-03-04T16:28:21.764+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:54.941476+0000 front 2024-03-04T16:27:54.941441+0000 (oldest deadline 2024-03-04T16:28:20.841339+0000) 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -250> 2024-03-04T16:28:22.741+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:46.840005+0000 front 2024-03-04T16:27:46.840065+0000 (oldest deadline 2024-03-04T16:28:12.139882+0000) 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: -249> 2024-03-04T16:28:22.741+0000 7f5916e58700 -1 osd.4 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:54.941476+0000 front 2024-03-04T16:27:54.941441+0000 (oldest deadline 2024-03-04T16:28:20.841339+0000) 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 0> 2024-03-04T16:28:24.076+0000 7f5905e36700 -1 *** Caught signal (Segmentation fault) ** 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: in thread 7f5905e36700 thread_name:safe_timer 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: ceph version 19.0.0-1826-g9ca0b5de (9ca0b5de35ed230e84387abdd8e619ed8e4f3282) squid (dev) 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5921cbad20] 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55ca57688cca] 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55ca57716b58] 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55ca572fbda3] 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55ca57587d01] 2024-03-04T16:28:24.201 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55ca57276974] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55ca572789ca] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55ca5727a933] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55ca571dc56a] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55ca57836fef] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55ca57838311] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f5921cb01ca] 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 13: clone() 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4[38540]: 2024-03-04T16:28:24.202 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 podman[156689]: 2024-03-04 16:28:24.196628326 +0000 UTC m=+5.688348396 container died 3ccb8d6150b8e9ae13e08fada95f97fc0d8283cae76db4fc12d4ed5ebf7ce04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-03-04T16:28:24.824 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:24.543+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:24.824 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:24.543+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:24.825 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:24.594+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:24.825 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:24.594+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:24.825 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 podman[156689]: 2024-03-04 16:28:24.582373061 +0000 UTC m=+6.074093136 container cleanup 3ccb8d6150b8e9ae13e08fada95f97fc0d8283cae76db4fc12d4ed5ebf7ce04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5) 2024-03-04T16:28:24.825 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 bash[156689]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4 2024-03-04T16:28:25.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 podman[156709]: 2024-03-04 16:28:24.825203053 +0000 UTC m=+0.621762761 container remove 3ccb8d6150b8e9ae13e08fada95f97fc0d8283cae76db4fc12d4ed5ebf7ce04f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0) 2024-03-04T16:28:25.108 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:24 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.4.service: Main process exited, code=exited, status=139/n/a 2024-03-04T16:28:25.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:24.878+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:25.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:24 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:24.878+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:25.470 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.126316263 +0000 UTC m=+0.107212840 container create 213c20124b9d5064aa634a968170446e1bd4e8b41e2fa16b42e34b209c07630c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:28:25.470 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.037776731 +0000 UTC m=+0.018673263 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:25.470 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.33681507 +0000 UTC m=+0.317711602 container init 213c20124b9d5064aa634a968170446e1bd4e8b41e2fa16b42e34b209c07630c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222) 2024-03-04T16:28:25.470 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.339989415 +0000 UTC m=+0.320885947 container start 213c20124b9d5064aa634a968170446e1bd4e8b41e2fa16b42e34b209c07630c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2) 2024-03-04T16:28:25.470 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.470459815 +0000 UTC m=+0.451356360 container attach 213c20124b9d5064aa634a968170446e1bd4e8b41e2fa16b42e34b209c07630c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-04T16:28:25.857 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:25.571+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:25.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:25.571+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:25.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:25.600+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:25.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:25.600+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:26.256 INFO:journalctl@ceph.osd.4.smithi176.stdout:Mar 04 16:28:25 smithi176 podman[156797]: 2024-03-04 16:28:25.968114655 +0000 UTC m=+0.949011190 container died 213c20124b9d5064aa634a968170446e1bd4e8b41e2fa16b42e34b209c07630c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240222) 2024-03-04T16:28:26.257 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:25.915+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:26.257 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:25 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:25.915+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:26.583 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.4.service' 2024-03-04T16:28:26.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:26.633 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-04T16:28:26.633 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-04T16:28:26.633 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.5 2024-03-04T16:28:26.858 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 systemd[1]: Stopping Ceph osd.5 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:26.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:26.545+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:26.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:26.546+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:26.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:26.589+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:26.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:26.589+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:27.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:26.921+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:27.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:26.921+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:27.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:26.978+0000 7f2316942700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:27.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:26.978+0000 7f2316942700 -1 osd.5 521 *** Got signal Terminated *** 2024-03-04T16:28:27.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:26 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:26.978+0000 7f2316942700 -1 osd.5 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:27.805 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:27.530+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:27.805 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:27.530+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:27.805 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:27.618+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:27.806 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:27.618+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:28.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:27.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:28.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:27.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:28.108 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:27 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:27.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:02.583259+0000 front 2024-03-04T16:28:02.583427+0000 (oldest deadline 2024-03-04T16:28:27.883316+0000) 2024-03-04T16:28:28.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:28.528+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:28.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:28.528+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:28.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:28.658+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:28.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:28.658+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:29.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:28.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:29.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:28.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:29.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:28 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:28.933+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:02.583259+0000 front 2024-03-04T16:28:02.583427+0000 (oldest deadline 2024-03-04T16:28:27.883316+0000) 2024-03-04T16:28:29.857 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:29.511+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:29.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:29.511+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:29.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:29.631+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:29.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:29.631+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:30.361 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:29.903+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:30.361 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:29.903+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:30.361 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:29 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:29.903+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:02.583259+0000 front 2024-03-04T16:28:02.583427+0000 (oldest deadline 2024-03-04T16:28:27.883316+0000) 2024-03-04T16:28:30.857 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:30.674+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:30.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:30.674+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:30.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:30.512+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:30.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:30.512+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:30.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:30.512+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:31.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:30.899+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:31.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:30.899+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:31.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:30 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:30.899+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:02.583259+0000 front 2024-03-04T16:28:02.583427+0000 (oldest deadline 2024-03-04T16:28:27.883316+0000) 2024-03-04T16:28:31.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:31.721+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:31.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:31.721+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:31.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:31.554+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:31.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:31.554+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:31.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:31.554+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:32.357 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:31.864+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.581237+0000 front 2024-03-04T16:27:45.581366+0000 (oldest deadline 2024-03-04T16:28:11.481176+0000) 2024-03-04T16:28:32.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:31.864+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.782371+0000 front 2024-03-04T16:27:56.782310+0000 (oldest deadline 2024-03-04T16:28:20.882277+0000) 2024-03-04T16:28:32.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:31 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5[44192]: 2024-03-04T16:28:31.864+0000 7f230ef2e700 -1 osd.5 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:02.583259+0000 front 2024-03-04T16:28:02.583427+0000 (oldest deadline 2024-03-04T16:28:27.883316+0000) 2024-03-04T16:28:32.743 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:32 smithi176 podman[156904]: 2024-03-04 16:28:32.47740707 +0000 UTC m=+5.631859377 container died fa5ecd04145e4f19a4f9fa5fce3e9c976116b61b7641bd76e42d84d7c45a4003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-04T16:28:32.743 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:32.599+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:32.743 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:32.599+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:32.743 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:32.599+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:33.061 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:32 smithi176 podman[156904]: 2024-03-04 16:28:32.868935544 +0000 UTC m=+6.023387856 container cleanup fa5ecd04145e4f19a4f9fa5fce3e9c976116b61b7641bd76e42d84d7c45a4003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, GIT_CLEAN=True) 2024-03-04T16:28:33.061 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:32 smithi176 bash[156904]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5 2024-03-04T16:28:33.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:32.743+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:33.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:32.743+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:33.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:32 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:32.743+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:33.358 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[156932]: 2024-03-04 16:28:33.061847576 +0000 UTC m=+0.584141768 container remove fa5ecd04145e4f19a4f9fa5fce3e9c976116b61b7641bd76e42d84d7c45a4003 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240222, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-04T16:28:33.648 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:33.601+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:33.648 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:33.601+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:33.648 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:33.601+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:33.648 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.379447894 +0000 UTC m=+0.099448413 container create 0d78e3b42d9dec4d68370deaf5f0ab6f76f4b63151e81059a1badb199463c99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-04T16:28:33.649 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.29834351 +0000 UTC m=+0.018344022 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:33.649 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.573830165 +0000 UTC m=+0.293830680 container init 0d78e3b42d9dec4d68370deaf5f0ab6f76f4b63151e81059a1badb199463c99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240222) 2024-03-04T16:28:33.649 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.576872388 +0000 UTC m=+0.296872899 container start 0d78e3b42d9dec4d68370deaf5f0ab6f76f4b63151e81059a1badb199463c99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d) 2024-03-04T16:28:33.649 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.648786112 +0000 UTC m=+0.368786637 container attach 0d78e3b42d9dec4d68370deaf5f0ab6f76f4b63151e81059a1badb199463c99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, ceph=True, org.label-schema.license=GPLv2) 2024-03-04T16:28:34.107 INFO:journalctl@ceph.osd.5.smithi176.stdout:Mar 04 16:28:33 smithi176 podman[157023]: 2024-03-04 16:28:33.870567568 +0000 UTC m=+0.590568077 container died 0d78e3b42d9dec4d68370deaf5f0ab6f76f4b63151e81059a1badb199463c99a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-04T16:28:34.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:33.763+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:34.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:33.763+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:34.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:33 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:33.763+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:34.460 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.5.service' 2024-03-04T16:28:34.503 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:34.503 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-04T16:28:34.504 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-04T16:28:34.504 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.6 2024-03-04T16:28:34.727 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.555+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:34.727 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.555+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:34.727 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.555+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:34.727 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 systemd[1]: Stopping Ceph osd.6 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:35.107 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:34.727+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:35.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:34.727+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:35.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:34.727+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:35.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.830+0000 7f6b5dd85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:35.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.830+0000 7f6b5dd85700 -1 osd.6 521 *** Got signal Terminated *** 2024-03-04T16:28:35.108 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:34 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:34.830+0000 7f6b5dd85700 -1 osd.6 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:35.857 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:35.529+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:35.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:35.529+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:35.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:35.529+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:35.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:35.726+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:35.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:35.726+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:35.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:35 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:35.726+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:36.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:36.773+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:36.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:36.773+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:36.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:36.773+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:36.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:36.537+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:36.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:36.537+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:36.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:36 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:36.537+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:37.565+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:37.565+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:37.565+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:37.565+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.789111+0000 front 2024-03-04T16:28:12.789017+0000 (oldest deadline 2024-03-04T16:28:37.389268+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:37.790+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:37.790+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:37.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:37.790+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:37.859 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:37 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:37.790+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:38.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:38.563+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:38.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:38.563+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:38.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:38.563+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:38.859 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:38.563+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.789111+0000 front 2024-03-04T16:28:12.789017+0000 (oldest deadline 2024-03-04T16:28:37.389268+0000) 2024-03-04T16:28:38.860 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:38.796+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:38.860 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:38.796+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:38.860 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:38.796+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:38.860 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:38 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:38.796+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:39.562+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:44.284971+0000 front 2024-03-04T16:27:44.284886+0000 (oldest deadline 2024-03-04T16:28:10.184846+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:39.562+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:56.586375+0000 front 2024-03-04T16:27:56.586351+0000 (oldest deadline 2024-03-04T16:28:19.486223+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:39.562+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:05.287707+0000 front 2024-03-04T16:28:05.287775+0000 (oldest deadline 2024-03-04T16:28:30.488141+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6[49764]: 2024-03-04T16:28:39.562+0000 7f6b56371700 -1 osd.6 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.789111+0000 front 2024-03-04T16:28:12.789017+0000 (oldest deadline 2024-03-04T16:28:37.389268+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:39.804+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:39.804+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:39.804+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:39.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:39 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:39.804+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:40.546 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:40 smithi176 podman[157130]: 2024-03-04 16:28:40.224066619 +0000 UTC m=+5.491252001 container died 9e1430724f102db9514a2164e3387a56b3fd0bb29de45275179e6e3281c52a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:28:40.546 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:40 smithi176 podman[157130]: 2024-03-04 16:28:40.54664495 +0000 UTC m=+5.813830332 container cleanup 9e1430724f102db9514a2164e3387a56b3fd0bb29de45275179e6e3281c52a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-04T16:28:40.843 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:40 smithi176 bash[157130]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6 2024-03-04T16:28:40.843 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:40 smithi176 podman[157151]: 2024-03-04 16:28:40.73946607 +0000 UTC m=+0.513358223 container remove 9e1430724f102db9514a2164e3387a56b3fd0bb29de45275179e6e3281c52a66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6, org.label-schema.build-date=20240222, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-04T16:28:41.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:40.843+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:41.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:40.843+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:41.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:40.843+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:41.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:40 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:40.843+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:41.363 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.068722596 +0000 UTC m=+0.018825518 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:41.363 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.207423689 +0000 UTC m=+0.157526596 container create 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:28:41.718 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.426907301 +0000 UTC m=+0.377010213 container init 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8) 2024-03-04T16:28:41.718 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.429940848 +0000 UTC m=+0.380043757 container start 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-04T16:28:41.718 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.526677275 +0000 UTC m=+0.476780185 container attach 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240222, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-04T16:28:41.719 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:28:41.364Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.38:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.38:8765: connect: connection refused" 2024-03-04T16:28:41.719 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:28:41.366Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.38:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.38:8765: connect: connection refused" 2024-03-04T16:28:41.719 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:28:41.367Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.38:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.38:8765: connect: connection refused" 2024-03-04T16:28:41.719 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:28:41.370Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.38:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.38:8765: connect: connection refused" 2024-03-04T16:28:41.719 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:28:41.370Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.38:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.38:8765: connect: connection refused" 2024-03-04T16:28:42.062 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:41 smithi176 podman[157245]: 2024-03-04 16:28:41.719068483 +0000 UTC m=+0.669171397 container died 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-04T16:28:42.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:41.877+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:42.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:41.877+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:42.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:41.877+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:42.062 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:41 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:41.877+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:42.303 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.6.service' 2024-03-04T16:28:42.331 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:42 smithi176 podman[157271]: 2024-03-04 16:28:42.280396989 +0000 UTC m=+0.555549435 container remove 2e9e26893a52aca886a799d1ee2dc1771fe5440b5726655fda9dd3232ee153f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, org.label-schema.vendor=CentOS, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:28:42.331 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:42 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-04T16:28:42.332 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:42 smithi176 systemd[1]: Stopped Ceph osd.6 for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:28:42.332 INFO:journalctl@ceph.osd.6.smithi176.stdout:Mar 04 16:28:42 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.6.service: Consumed 11min 27.818s CPU time. 2024-03-04T16:28:42.348 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:42.348 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-04T16:28:42.349 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-04T16:28:42.349 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.7 2024-03-04T16:28:42.607 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 systemd[1]: Stopping Ceph osd.7 for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:43.107 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.717+0000 7f49168ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.717+0000 7f49168ba700 -1 osd.7 521 *** Got signal Terminated *** 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.717+0000 7f49168ba700 -1 osd.7 521 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.863+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.863+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.863+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:43.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:42 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:42.863+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:44.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:43.852+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:44.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:43.852+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:44.109 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:43.852+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:44.109 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:43 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:43.852+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:45.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:44 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:44.873+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:45.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:44 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:44.873+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:45.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:44 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:44.873+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:45.358 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:44 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:44.873+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:46.107 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:45 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:45.838+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:46.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:45 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:45.838+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:46.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:45 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:45.838+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:46.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:45 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:45.838+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:47.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:46 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:46.789+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6803 osd.0 since back 2024-03-04T16:27:45.770360+0000 front 2024-03-04T16:27:45.770326+0000 (oldest deadline 2024-03-04T16:28:12.770648+0000) 2024-03-04T16:28:47.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:46 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:46.789+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6807 osd.1 since back 2024-03-04T16:27:58.472924+0000 front 2024-03-04T16:27:58.473017+0000 (oldest deadline 2024-03-04T16:28:21.972827+0000) 2024-03-04T16:28:47.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:46 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:46.789+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6811 osd.2 since back 2024-03-04T16:28:06.673931+0000 front 2024-03-04T16:28:06.673918+0000 (oldest deadline 2024-03-04T16:28:32.573778+0000) 2024-03-04T16:28:47.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:46 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:46.789+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.38:6815 osd.3 since back 2024-03-04T16:28:12.574235+0000 front 2024-03-04T16:28:12.574225+0000 (oldest deadline 2024-03-04T16:28:37.274210+0000) 2024-03-04T16:28:47.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:46 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7[55371]: 2024-03-04T16:28:46.789+0000 7f490eea6700 -1 osd.7 521 heartbeat_check: no reply from 172.21.15.176:6802 osd.4 since back 2024-03-04T16:28:20.774967+0000 front 2024-03-04T16:28:20.775065+0000 (oldest deadline 2024-03-04T16:28:46.074981+0000) 2024-03-04T16:28:48.500 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:48 smithi176 podman[157351]: 2024-03-04 16:28:48.211308174 +0000 UTC m=+5.632662542 container died c4f7b783444d7e30b6ec26a78d728f456830408236a3e3d9d98d5dbfac877093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-04T16:28:48.800 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:48 smithi176 podman[157351]: 2024-03-04 16:28:48.624736048 +0000 UTC m=+6.046090410 container cleanup c4f7b783444d7e30b6ec26a78d728f456830408236a3e3d9d98d5dbfac877093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-04T16:28:48.800 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:48 smithi176 bash[157351]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7 2024-03-04T16:28:49.108 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:48 smithi176 podman[157378]: 2024-03-04 16:28:48.800752868 +0000 UTC m=+0.593023156 container remove c4f7b783444d7e30b6ec26a78d728f456830408236a3e3d9d98d5dbfac877093 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-04T16:28:49.481 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.209716624 +0000 UTC m=+0.112764883 container create 418708d5456726cf29685770bd66041168a9db06106307435aa4c7ba6ff0efb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222) 2024-03-04T16:28:49.482 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.1149903 +0000 UTC m=+0.018038573 image pull 0d1497d8ab5e9dc8a566513caa4bcfff51798ecc87de6dcca9e525d8d416d692 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977 2024-03-04T16:28:49.482 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.412145736 +0000 UTC m=+0.315193997 container init 418708d5456726cf29685770bd66041168a9db06106307435aa4c7ba6ff0efb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-03-04T16:28:49.482 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.415095439 +0000 UTC m=+0.318143702 container start 418708d5456726cf29685770bd66041168a9db06106307435aa4c7ba6ff0efb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240222, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-04T16:28:49.482 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.479002189 +0000 UTC m=+0.382050465 container attach 418708d5456726cf29685770bd66041168a9db06106307435aa4c7ba6ff0efb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-04T16:28:49.858 INFO:journalctl@ceph.osd.7.smithi176.stdout:Mar 04 16:28:49 smithi176 podman[157469]: 2024-03-04 16:28:49.705975081 +0000 UTC m=+0.609023333 container died 418708d5456726cf29685770bd66041168a9db06106307435aa4c7ba6ff0efb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240222, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, CEPH_POINT_RELEASE=, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-04T16:28:50.284 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@osd.7.service' 2024-03-04T16:28:50.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:28:50.328 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-04T16:28:50.328 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-04T16:28:50.328 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@rgw.foo.a 2024-03-04T16:28:50.631 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:28:50 smithi038 systemd[1]: Stopping Ceph rgw.foo.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:28:50.898 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:28:50 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a[65209]: 2024-03-04T16:28:50.631+0000 7f47d5e3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-04T16:28:50.899 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:28:50 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a[65209]: 2024-03-04T16:28:50.631+0000 7f47e46b2ac0 -1 shutting down 2024-03-04T16:29:00.967 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:29:00 smithi038 bash[213763]: time="2024-03-04T16:29:00Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-04T16:29:00.967 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:29:00 smithi038 podman[213763]: 2024-03-04 16:29:00.648637788 +0000 UTC m=+10.121975577 container died cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240222, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-04T16:29:01.242 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:29:01 smithi038 podman[213763]: 2024-03-04 16:29:01.173163125 +0000 UTC m=+10.646500912 container cleanup cebe9a020f5b3e1d0857204d6e8b63d6429921d13033e0655e27a2152c46cd6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240222, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:29:01.242 INFO:journalctl@ceph.rgw.foo.a.smithi038.stdout:Mar 04 16:29:01 smithi038 bash[213763]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-rgw-foo-a 2024-03-04T16:29:01.542 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@rgw.foo.a.service' 2024-03-04T16:29:01.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:29:01.597 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-04T16:29:01.597 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-04T16:29:01.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-c453baf0-da3f-11ee-95c6-87774f69a715@prometheus.a 2024-03-04T16:29:01.900 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 systemd[1]: Stopping Ceph prometheus.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.900Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-04T16:29:02.192 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.901Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.901Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.907Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.908Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.908Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a[69488]: ts=2024-03-04T16:29:01.908Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-04T16:29:02.193 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:01 smithi176 podman[157574]: 2024-03-04 16:29:01.910855318 +0000 UTC m=+0.117460194 container died 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:02.575 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:02 smithi176 podman[157574]: 2024-03-04 16:29:02.283708855 +0000 UTC m=+0.490313773 container cleanup 27e618f9e250597e5cbc1fb124ae2ebb500b2e5709d4a65973e0d74d890cb001 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:02.575 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Mar 04 16:29:02 smithi176 bash[157574]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-prometheus-a 2024-03-04T16:29:02.682 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c453baf0-da3f-11ee-95c6-87774f69a715@prometheus.a.service' 2024-03-04T16:29:02.719 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-04T16:29:02.719 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-04T16:29:02.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c453baf0-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:29:02.918 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T16:29:05.421 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:05 smithi038 systemd[1]: Stopping Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:05.742 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:05 smithi038 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a[82794]: ts=2024-03-04T16:29:05.420Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-04T16:29:05.742 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:05 smithi038 podman[214269]: 2024-03-04 16:29:05.461684862 +0000 UTC m=+0.130996679 container died 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:06.242 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:05 smithi038 podman[214269]: 2024-03-04 16:29:05.976034019 +0000 UTC m=+0.645345875 container cleanup 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:06.243 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:05 smithi038 bash[214269]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a 2024-03-04T16:29:06.637 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:06 smithi038 podman[214283]: 2024-03-04 16:29:06.372921939 +0000 UTC m=+0.892959585 container remove 22fa2e1d5061fc63f6230593fa74fe6af62cb006e530f3e0baa332bab0bd8411 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:06.638 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:06 smithi038 podman[214283]: 2024-03-04 16:29:06.466320971 +0000 UTC m=+0.986358617 volume remove b64903e08967d3d9b0b7b0f6d1f7d1fe1f6ea2ba51e64e8fa6331333be2e6b90 2024-03-04T16:29:06.996 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:06 smithi038 systemd[1]: Stopping Ceph node-exporter.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:06.997 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:06 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-04T16:29:06.997 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:06 smithi038 systemd[1]: Stopped Ceph alertmanager.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:29:06.997 INFO:journalctl@ceph.alertmanager.a.smithi038.stdout:Mar 04 16:29:06 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@alertmanager.a.service: Consumed 1.751s CPU time. 2024-03-04T16:29:07.443 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 podman[214396]: 2024-03-04 16:29:07.11255521 +0000 UTC m=+0.109144188 container died aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:07.742 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 podman[214396]: 2024-03-04 16:29:07.44373125 +0000 UTC m=+0.440320237 container cleanup aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:07.742 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 bash[214396]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a 2024-03-04T16:29:07.743 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 podman[214410]: 2024-03-04 16:29:07.670045982 +0000 UTC m=+0.556191603 container remove aeca62e0f963391abe3159b4133432edec98251fcfbcc67c61acc56f42f25776 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-04T16:29:07.743 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:29:08.187 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-04T16:29:08.187 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 systemd[1]: Stopped Ceph node-exporter.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:29:08.187 INFO:journalctl@ceph.node-exporter.a.smithi038.stdout:Mar 04 16:29:07 smithi038 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.a.service: Consumed 4.833s CPU time. 2024-03-04T16:29:08.641 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c453baf0-da3f-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-04T16:29:08.834 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T16:29:10.858 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:10 smithi176 systemd[1]: Stopping Ceph iscsi.iscsi.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:11.357 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:11 smithi176 ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a[60521]: debug Shutdown received 2024-03-04T16:29:21.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 bash[157988]: time="2024-03-04T16:29:21Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-04T16:29:21.358 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 podman[157988]: 2024-03-04 16:29:21.061273281 +0000 UTC m=+10.124703229 container died 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240222, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, maintainer=Guillaume Abrioux ) 2024-03-04T16:29:21.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 podman[157988]: 2024-03-04 16:29:21.586799878 +0000 UTC m=+10.650229816 container cleanup 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240222) 2024-03-04T16:29:21.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 bash[157988]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a 2024-03-04T16:29:22.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 podman[158015]: 2024-03-04 16:29:21.804820869 +0000 UTC m=+0.743507812 container remove 09831de96ff50e8b8307949f8464ccc28178a50d76e58d6945d404231d49a283 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:6d920225840c1daadea71eb62fa66076c3ca6088306e8e7302cea6cb04aef977, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-iscsi-iscsi-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=918ebec1af6f745255c974e3941e386838a94fd8, RELEASE=wip-asm0deuz-testing-2024-03-04-1211-9ca0b5d, org.label-schema.build-date=20240222, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-04T16:29:22.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:21 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-04T16:29:22.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:22 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-04T16:29:22.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:22 smithi176 systemd[1]: Stopped Ceph iscsi.iscsi.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:29:22.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi176.stdout:Mar 04 16:29:22 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@iscsi.iscsi.a.service: Consumed 3.583s CPU time. 2024-03-04T16:29:23.609 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:23 smithi176 systemd[1]: Stopping Ceph grafana.a for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:24.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:23 smithi176 podman[158278]: 2024-03-04 16:29:23.789834646 +0000 UTC m=+0.122868311 container died e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, name=ubi8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8.5, description=Ceph Grafana Container, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-04-29T11:48:11.055897) 2024-03-04T16:29:24.359 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 podman[158278]: 2024-03-04 16:29:24.229132467 +0000 UTC m=+0.562166171 container cleanup e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, release=239.1651231664, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , name=ubi8, version=8.5, vcs-type=git, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0) 2024-03-04T16:29:24.359 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 bash[158278]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a 2024-03-04T16:29:24.843 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 podman[158292]: 2024-03-04 16:29:24.446960796 +0000 UTC m=+0.659577593 container remove e4287ee36f44730e15144e43a23f0cf5058b05b28297961ce798e9c07087dd64 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-grafana-a, version=8.5, build-date=2022-04-29T11:48:11.055897, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base rhel8, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, maintainer=Paul Cuzner , name=ubi8, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container) 2024-03-04T16:29:24.843 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:29:24.843 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-04T16:29:24.843 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 systemd[1]: Stopped Ceph grafana.a for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:29:24.843 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Mar 04 16:29:24 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@grafana.a.service: Consumed 13.417s CPU time. 2024-03-04T16:29:25.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:24 smithi176 systemd[1]: Stopping Ceph node-exporter.b for c453baf0-da3f-11ee-95c6-87774f69a715... 2024-03-04T16:29:25.366 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 podman[158405]: 2024-03-04 16:29:25.110224338 +0000 UTC m=+0.107066264 container died e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:29:25.775 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 podman[158405]: 2024-03-04 16:29:25.550114885 +0000 UTC m=+0.546956822 container cleanup e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:29:25.775 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 bash[158405]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b 2024-03-04T16:29:26.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 podman[158419]: 2024-03-04 16:29:25.776273103 +0000 UTC m=+0.664253839 container remove e777ec577ed1463a646a24257c2bb6803da88577906ee52a6c312a16f1b489ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c453baf0-da3f-11ee-95c6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-04T16:29:26.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-04T16:29:26.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-04T16:29:26.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 systemd[1]: Stopped Ceph node-exporter.b for c453baf0-da3f-11ee-95c6-87774f69a715. 2024-03-04T16:29:26.107 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Mar 04 16:29:25 smithi176 systemd[1]: ceph-c453baf0-da3f-11ee-95c6-87774f69a715@node-exporter.b.service: Consumed 4.581s CPU time. 2024-03-04T16:29:26.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:29:26.693 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-04T16:29:26.722 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-04T16:29:26.725 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi038/crash 2024-03-04T16:29:26.726 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:29:26.803 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:29:26.803 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:29:26.805 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi176/crash 2024-03-04T16:29:26.806 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash -- . 2024-03-04T16:29:26.858 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-04T16:29:26.858 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2024-03-04T16:29:26.859 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-04T16:29:26.860 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-04T16:29:26.967 INFO:teuthology.orchestra.run.smithi038.stdout:2024-03-04T16:10:00.001602+0000 mon.a (mon.0) 1063 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-04T16:29:27.022 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-04T16:29:27.022 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[SEC\]' /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-04T16:29:27.053 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]' /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-04T16:29:27.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[WRN\]' /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-04T16:29:27.194 INFO:teuthology.orchestra.run.smithi038.stdout:2024-03-04T16:10:00.001602+0000 mon.a (mon.0) 1063 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-04T16:29:27.195 INFO:tasks.cephadm:Compressing logs... 2024-03-04T16:29:27.195 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:29:27.239 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-04T16:29:27.289 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-04T16:29:27.289 INFO:teuthology.orchestra.run.smithi176.stderr:: No such file or directory 2024-03-04T16:29:27.300 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:29:27.301 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-04T16:29:27.301 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-04T16:29:27.301 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-04T16:29:27.301 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-04T16:29:27.302 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/tcmu-runner.log 2024-03-04T16:29:27.303 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:29:27.303 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:29:27.303 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:29:27.303 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-04T16:29:27.303 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:29:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-04T16:29:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-04T16:29:27.315 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-04T16:29:27.315 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-04T16:29:27.315 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log: /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-04T16:29:27.315 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-04T16:29:27.315 INFO:teuthology.orchestra.run.smithi038.stderr: 57.8% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-04T16:29:27.318 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-04T16:29:27.327 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log: /var/log/ceph/cephadm.log: 91.6% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:29:27.399 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log: /var/log/ceph/cephadm.log: 81.4% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:29:27.444 INFO:teuthology.orchestra.run.smithi038.stderr: 88.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-04T16:29:27.462 INFO:teuthology.orchestra.run.smithi176.stderr: 67.2% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/tcmu-runner.log.gz 2024-03-04T16:29:27.504 INFO:teuthology.orchestra.run.smithi038.stderr: 91.1% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:29:27.504 INFO:teuthology.orchestra.run.smithi176.stderr: 92.0% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-04T16:29:27.530 INFO:teuthology.orchestra.run.smithi176.stderr: 89.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:29:27.752 INFO:teuthology.orchestra.run.smithi038.stderr: 93.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:29:27.805 INFO:teuthology.orchestra.run.smithi176.stderr: 91.2% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-04T16:29:27.982 INFO:teuthology.orchestra.run.smithi176.stderr: 89.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-04T16:29:28.011 INFO:teuthology.orchestra.run.smithi176.stderr: 85.0% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-04T16:29:29.214 INFO:teuthology.orchestra.run.smithi038.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-04T16:29:32.232 INFO:teuthology.orchestra.run.smithi038.stderr: 89.0% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-04T16:29:38.846 INFO:teuthology.orchestra.run.smithi176.stderr: 90.7% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-04T16:29:40.498 INFO:teuthology.orchestra.run.smithi038.stderr: 90.6% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-04T16:29:46.661 INFO:teuthology.orchestra.run.smithi038.stderr: 90.1% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-04T16:32:47.104 INFO:teuthology.orchestra.run.smithi176.stderr: 92.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-04T16:33:33.001 INFO:teuthology.orchestra.run.smithi176.stderr: 93.3% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-04T16:34:37.322 INFO:teuthology.orchestra.run.smithi176.stderr: 93.6% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-04T16:35:36.021 INFO:teuthology.orchestra.run.smithi176.stderr: 93.0% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-04T16:35:36.023 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-03-04T16:35:36.023 INFO:teuthology.orchestra.run.smithi176.stderr:real 6m8.779s 2024-03-04T16:35:36.023 INFO:teuthology.orchestra.run.smithi176.stderr:user 5m0.799s 2024-03-04T16:35:36.023 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m12.955s 2024-03-04T16:42:40.522 INFO:teuthology.orchestra.run.smithi038.stderr: 93.6% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-04T16:48:40.536 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-04T16:49:07.677 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-04T16:49:25.480 INFO:teuthology.orchestra.run.smithi038.stderr: 92.9% -- replaced with /var/log/ceph/c453baf0-da3f-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-04T16:49:25.502 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-04T16:49:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:real 19m58.270s 2024-03-04T16:49:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:user 7m51.892s 2024-03-04T16:49:25.502 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m21.818s 2024-03-04T16:49:25.502 INFO:tasks.cephadm:Archiving logs... 2024-03-04T16:49:25.503 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi038/log 2024-03-04T16:49:25.504 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:50:34.842 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi176/log 2024-03-04T16:50:34.843 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-04T16:51:24.583 INFO:tasks.cephadm:Removing cluster... 2024-03-04T16:51:24.584 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c453baf0-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:51:25.208 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T16:51:26.536 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c453baf0-da3f-11ee-95c6-87774f69a715 --force 2024-03-04T16:51:27.159 INFO:teuthology.orchestra.run.smithi176.stdout:Deleting cluster with fsid: c453baf0-da3f-11ee-95c6-87774f69a715 2024-03-04T16:51:27.854 INFO:tasks.cephadm:Removing cephadm ... 2024-03-04T16:51:27.854 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:51:27.890 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-04T16:51:27.905 INFO:tasks.cephadm:Teardown complete 2024-03-04T16:51:27.906 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-04T16:51:27.915 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-04T16:51:27.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T16:51:27.932 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-04T16:51:28.033 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-04T16:51:28.044 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-04T16:51:28.044 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:51:28.047 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-04T16:51:28.070 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:51:28.070 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 -1221us[-1227us] +/- 120ms 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -167us[ -173us] +/- 70ms 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 +889us[ +883us] +/- 66ms 2024-03-04T16:51:28.094 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:51:28.108 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-04T16:51:28.108 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-04T16:51:28.108 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 -1205us[-1205us] +/- 120ms 2024-03-04T16:51:28.108 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 15 -160us[ -171us] +/- 70ms 2024-03-04T16:51:28.109 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +896us[ +885us] +/- 67ms 2024-03-04T16:51:28.109 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-04T16:51:28.109 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-04T16:51:28.119 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-04T16:51:28.120 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-04T16:51:28.132 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:51:28.299 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-04T16:51:28.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:51:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:51:28.421 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-04T16:51:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:51:28.449 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-04T16:51:28.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-04T16:51:28.469 INFO:teuthology.task.internal:Duration was 3704.934858 seconds 2024-03-04T16:51:28.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-04T16:51:28.480 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-04T16:51:28.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:51:28.483 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-04T16:51:28.562 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:51:28.563 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-04T16:51:29.048 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-04T16:51:29.048 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-03-04T16:51:29.049 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:51:29.096 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-03-04T16:51:29.096 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-04T16:51:29.124 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-04T16:51:29.124 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:51:29.127 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-04T16:51:29.193 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-04T16:51:29.193 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:51:29.200 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-04T16:51:30.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-04T16:51:30.523 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-04T16:51:30.523 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:51:30.551 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-04T16:51:30.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-04T16:51:30.595 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:51:30.597 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:51:30.633 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-03-04T16:51:30.667 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-03-04T16:51:30.692 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:51:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:51:30.721 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-04T16:51:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-04T16:51:30.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-04T16:51:30.757 INFO:teuthology.task.internal:Transferring archived files... 2024-03-04T16:51:30.757 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi038 2024-03-04T16:51:30.757 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:51:30.826 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239/remote/smithi176 2024-03-04T16:51:30.827 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-04T16:51:30.866 INFO:teuthology.task.internal:Removing archive directory... 2024-03-04T16:51:30.866 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:51:30.869 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-04T16:51:30.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-04T16:51:30.926 INFO:teuthology.task.internal:Not uploading archives. 2024-03-04T16:51:30.926 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-04T16:51:30.959 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-04T16:51:30.960 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:51:30.962 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-04T16:51:30.974 INFO:teuthology.orchestra.run.smithi038.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 16:51 /home/ubuntu/cephtest 2024-03-04T16:51:30.976 INFO:teuthology.orchestra.run.smithi176.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 4 16:51 /home/ubuntu/cephtest 2024-03-04T16:51:30.981 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-04T16:51:30.994 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-04T16:51:31.018 INFO:teuthology.nuke:Checking targets against current locks 2024-03-04T16:51:31.043 DEBUG:teuthology.nuke:shortname: smithi038 2024-03-04T16:51:31.044 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:51:31.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:41:03.192480', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEWPHYoC11lbobcDd3OU4DfFN73nuWEPHzYCvyheTmedIbav+2IEE5DJFEM2LleEfu0glEDyCpNsSmbv82GUOJ4='} 2024-03-04T16:51:31.068 DEBUG:teuthology.nuke:shortname: smithi176 2024-03-04T16:51:31.069 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-04T16:51:31.089 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/tonay-2024-03-04_15:07:38-orch:cephadm-wip-asm0deuz-testing-2024-03-04-1211-distro-default-smithi/7581239', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-04 15:41:03.194042', 'locked_by': 'scheduled_tonay@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7TncKNCLhunUYsBSAMR0akyNySygcv1hw1IEgYtU9a4jDPcms819RdzNhFYLD7lXC6gyUKeoY3JFyhShtiDQk='} 2024-03-04T16:51:31.120 INFO:teuthology.orchestra.console.smithi038:Power off 2024-03-04T16:51:31.121 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:51:31.144 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-03-04T16:51:31.145 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:31.159 INFO:teuthology.orchestra.console.smithi176:Power off 2024-03-04T16:51:31.159 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-04T16:51:31.169 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-04T16:51:31.183 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-03-04T16:51:31.184 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:31.207 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-04T16:51:35.170 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:35.208 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:35.281 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-04T16:51:35.317 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-04T16:51:39.286 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:39.318 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:39.396 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-03-04T16:51:39.428 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-03-04T16:51:39.498 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-03-04T16:51:43.432 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-04T16:51:43.541 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-03-04T16:51:43.641 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-03-04T16:51:43.745 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3704.9348583221436 failure_reason: '"2024-03-04T16:10:00.001602+0000 mon.a (mon.0) 1063 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_tonay@teuthology success: false 2024-03-04T16:51:43.745 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-04T16:51:43.823 INFO:teuthology.run:FAIL